{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Datawhale智慧海洋建设-Task4模型建立"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此部分为智慧海洋建设竞赛的模型建立模块。在该模块中主要介绍了如何进行模型建立并对模型调优。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hide_input": true
   },
   "source": [
    "## 学习目标\n",
    "1. 学习如何选择合适的模型以及如何通过模型来进行特征选择\n",
    "2. 掌握随机森林、lightGBM、Xgboost模型的使用。\n",
    "3. 掌握贝叶斯优化方法的具体使用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 内容介绍\n",
    "1. 模型训练与预测\n",
    " - 随机森林\n",
    " - lightGBM模型\n",
    " - Xgboost模型\n",
    "2. 交叉验证\n",
    "3. 模型调参\n",
    "4. 智慧海洋数据集模型代码示例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 模型训练与预测"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型训练与预测的主要步骤为：  \n",
    "(1):导入需要的工具库  \n",
    "(2):对数据预处理，包括导入数据集、处理数据等操作，具体为缺失值处理、连续特征归一化、类别特征转换等  \n",
    "(3):训练模型。选择合适的机器学习模型，利用训练集对模型进行训练，达到最佳拟合效果。  \n",
    "(4):预测结果。将待预测的数据输入到训练好的模型中，得到预测的结果。\n",
    "\n",
    "下面进行几种常用的分类算法进行介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 随机森林分类\n",
    "[随机森林参数介绍](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier)  \n",
    "随机森林是通过集成学习的思想将多棵树集成的一种算法，基本单元是决策树，而它的本质属于机器学习的一个分支——集成学习。\n",
    "随机森林模型的主要优点是：在当前算法中，具有较好的准确率；能够有效地运行在大数据集上；能够处理具有高维特征的输入样本，而且不需要降维；能够评估各个特征在分类问题上的重要性；在生成过程中，能够获取到内部生成误差的一种无偏估计；对于缺省值问题也能够获得很好的结果。  \n",
    "\n",
    "使用sklearn调用随机森林分类树进行预测算法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import f1_score\n",
    "#数据集导入\n",
    "iris=datasets.load_iris()\n",
    "feature=iris.feature_names\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "#随机森林\n",
    "clf=RandomForestClassifier(n_estimators=200)\n",
    "train_X,test_X,train_y,test_y = train_test_split(X,y,test_size=0.1,random_state=5)\n",
    "clf.fit(train_X,train_y)\n",
    "test_pred=clf.predict(test_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']\n",
      "[0.09838896 0.01544017 0.34365936 0.5425115 ]\n"
     ]
    }
   ],
   "source": [
    "#特征的重要性查看\n",
    "print(str(feature)+'\\n'+str(clf.feature_importances_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "采用F1 score进行模型的评价，[此为一篇csdn中对该评价方法的简单说明](https://blog.csdn.net/qq_14997473/article/details/82684300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "随机森林-macro： 0.818181818181818\n",
      "随机森林-weighted： 0.8\n"
     ]
    }
   ],
   "source": [
    "#F1-score 用于模型评价\n",
    "#如果是二分类问题则选择参数‘binary’\n",
    "#如果考虑类别的不平衡性，需要计算类别的加权平均，则使用‘weighted’\n",
    "#如果不考虑类别的不平衡性，计算宏平均，则使用‘macro’\n",
    "score=f1_score(test_y,test_pred,average='macro')\n",
    "print(\"随机森林-macro：\",score)\n",
    "score=f1_score(test_y,test_pred,average='weighted')\n",
    "print(\"随机森林-weighted：\",score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## lightGBM模型\n",
    "[lightGBM的学习可参见这篇文章](https://mp.weixin.qq.com/s/64xfT9WIgF3yEExpSxyshQ)  \n",
    "[lightGBM中文文档](https://lightgbm.apachecn.org/#/)这个对超参数的讲解较为详细，建议仔细阅读"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. lightGBM过拟合处理方案：\n",
    "- 使用较小的 max_bin\n",
    "- 使用较小的 num_leaves\n",
    "- 使用 min_data_in_leaf 和 min_sum_hessian_in_leaf\n",
    "- 通过设置 bagging_fraction 和 bagging_freq 来使用 bagging\n",
    "- 通过设置 feature_fraction 来使用特征子抽样\n",
    "- 使用更大的训练数据\n",
    "- 使用 lambda_l1, lambda_l2 和 min_gain_to_split 来使用正则\n",
    "- 尝试 max_depth 来避免生成过深的树  \n",
    "2. lightGBM针对更快的训练速度的解决方案\n",
    "- 通过设置 bagging_fraction 和 bagging_freq 参数来使用 bagging 方法\n",
    "- 通过设置 feature_fraction 参数来使用特征的子抽样\n",
    "- 使用较小的 max_bin\n",
    "- 使用 save_binary 在未来的学习过程对数据加载进行加速\n",
    "- 使用并行学习, 可参考 并行学习指南\n",
    "3. lightGBM针对更好的准确率\n",
    "- 使用较大的 max_bin （学习速度可能变慢）\n",
    "- 使用较小的 learning_rate 和较大的 num_iterations\n",
    "- 使用较大的 num_leaves （可能导致过拟合）\n",
    "- 使用更大的训练数据\n",
    "- 尝试 dart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\ttrain's multi_logloss: 0.975702\tvalidate's multi_logloss: 1.009\n",
      "[2]\ttrain's multi_logloss: 0.877457\tvalidate's multi_logloss: 0.914377\n",
      "[3]\ttrain's multi_logloss: 0.794798\tvalidate's multi_logloss: 0.824134\n",
      "[4]\ttrain's multi_logloss: 0.723326\tvalidate's multi_logloss: 0.750893\n",
      "[5]\ttrain's multi_logloss: 0.661667\tvalidate's multi_logloss: 0.682191\n",
      "[6]\ttrain's multi_logloss: 0.607721\tvalidate's multi_logloss: 0.628136\n",
      "[7]\ttrain's multi_logloss: 0.560519\tvalidate's multi_logloss: 0.574289\n",
      "[8]\ttrain's multi_logloss: 0.518687\tvalidate's multi_logloss: 0.529814\n",
      "[9]\ttrain's multi_logloss: 0.481561\tvalidate's multi_logloss: 0.485778\n",
      "[10]\ttrain's multi_logloss: 0.448635\tvalidate's multi_logloss: 0.449967\n",
      "[11]\ttrain's multi_logloss: 0.418864\tvalidate's multi_logloss: 0.414047\n",
      "[12]\ttrain's multi_logloss: 0.392319\tvalidate's multi_logloss: 0.386407\n",
      "[13]\ttrain's multi_logloss: 0.368389\tvalidate's multi_logloss: 0.357079\n",
      "[14]\ttrain's multi_logloss: 0.346782\tvalidate's multi_logloss: 0.33318\n",
      "[15]\ttrain's multi_logloss: 0.327196\tvalidate's multi_logloss: 0.308858\n",
      "[16]\ttrain's multi_logloss: 0.309539\tvalidate's multi_logloss: 0.288072\n",
      "[17]\ttrain's multi_logloss: 0.293482\tvalidate's multi_logloss: 0.268706\n",
      "[18]\ttrain's multi_logloss: 0.278991\tvalidate's multi_logloss: 0.25158\n",
      "[19]\ttrain's multi_logloss: 0.265753\tvalidate's multi_logloss: 0.23781\n",
      "[20]\ttrain's multi_logloss: 0.253744\tvalidate's multi_logloss: 0.226251\n",
      "[21]\ttrain's multi_logloss: 0.242663\tvalidate's multi_logloss: 0.211902\n",
      "[22]\ttrain's multi_logloss: 0.232649\tvalidate's multi_logloss: 0.202371\n",
      "[23]\ttrain's multi_logloss: 0.223439\tvalidate's multi_logloss: 0.192921\n",
      "[24]\ttrain's multi_logloss: 0.215001\tvalidate's multi_logloss: 0.182008\n",
      "[25]\ttrain's multi_logloss: 0.2072\tvalidate's multi_logloss: 0.175881\n",
      "[26]\ttrain's multi_logloss: 0.200111\tvalidate's multi_logloss: 0.168868\n",
      "[27]\ttrain's multi_logloss: 0.193543\tvalidate's multi_logloss: 0.160918\n",
      "[28]\ttrain's multi_logloss: 0.187559\tvalidate's multi_logloss: 0.155117\n",
      "[29]\ttrain's multi_logloss: 0.182121\tvalidate's multi_logloss: 0.148551\n",
      "[30]\ttrain's multi_logloss: 0.177063\tvalidate's multi_logloss: 0.141508\n",
      "[31]\ttrain's multi_logloss: 0.172155\tvalidate's multi_logloss: 0.136823\n",
      "[32]\ttrain's multi_logloss: 0.167851\tvalidate's multi_logloss: 0.13318\n",
      "[33]\ttrain's multi_logloss: 0.163832\tvalidate's multi_logloss: 0.127932\n",
      "[34]\ttrain's multi_logloss: 0.160045\tvalidate's multi_logloss: 0.124999\n",
      "[35]\ttrain's multi_logloss: 0.156511\tvalidate's multi_logloss: 0.11994\n",
      "[36]\ttrain's multi_logloss: 0.153185\tvalidate's multi_logloss: 0.117388\n",
      "[37]\ttrain's multi_logloss: 0.150086\tvalidate's multi_logloss: 0.113542\n",
      "[38]\ttrain's multi_logloss: 0.147138\tvalidate's multi_logloss: 0.11118\n",
      "[39]\ttrain's multi_logloss: 0.144376\tvalidate's multi_logloss: 0.107657\n",
      "[40]\ttrain's multi_logloss: 0.141792\tvalidate's multi_logloss: 0.105666\n",
      "[41]\ttrain's multi_logloss: 0.139327\tvalidate's multi_logloss: 0.102515\n",
      "[42]\ttrain's multi_logloss: 0.137023\tvalidate's multi_logloss: 0.101176\n",
      "[43]\ttrain's multi_logloss: 0.134844\tvalidate's multi_logloss: 0.0975092\n",
      "[44]\ttrain's multi_logloss: 0.132768\tvalidate's multi_logloss: 0.0948682\n",
      "[45]\ttrain's multi_logloss: 0.130798\tvalidate's multi_logloss: 0.0939896\n",
      "[46]\ttrain's multi_logloss: 0.128917\tvalidate's multi_logloss: 0.0915695\n",
      "[47]\ttrain's multi_logloss: 0.127132\tvalidate's multi_logloss: 0.0906398\n",
      "[48]\ttrain's multi_logloss: 0.12546\tvalidate's multi_logloss: 0.0892012\n",
      "[49]\ttrain's multi_logloss: 0.123835\tvalidate's multi_logloss: 0.0884964\n",
      "[50]\ttrain's multi_logloss: 0.122284\tvalidate's multi_logloss: 0.087185\n",
      "[51]\ttrain's multi_logloss: 0.120772\tvalidate's multi_logloss: 0.0849336\n",
      "[52]\ttrain's multi_logloss: 0.119346\tvalidate's multi_logloss: 0.0835437\n",
      "[53]\ttrain's multi_logloss: 0.11795\tvalidate's multi_logloss: 0.0829754\n",
      "[54]\ttrain's multi_logloss: 0.116534\tvalidate's multi_logloss: 0.0819892\n",
      "[55]\ttrain's multi_logloss: 0.115189\tvalidate's multi_logloss: 0.0808175\n",
      "[56]\ttrain's multi_logloss: 0.113915\tvalidate's multi_logloss: 0.0791856\n",
      "[57]\ttrain's multi_logloss: 0.112663\tvalidate's multi_logloss: 0.0778838\n",
      "[58]\ttrain's multi_logloss: 0.111477\tvalidate's multi_logloss: 0.0767819\n",
      "[59]\ttrain's multi_logloss: 0.110319\tvalidate's multi_logloss: 0.0761175\n",
      "[60]\ttrain's multi_logloss: 0.109189\tvalidate's multi_logloss: 0.075811\n",
      "[61]\ttrain's multi_logloss: 0.108108\tvalidate's multi_logloss: 0.0743217\n",
      "[62]\ttrain's multi_logloss: 0.107049\tvalidate's multi_logloss: 0.0730824\n",
      "[63]\ttrain's multi_logloss: 0.106037\tvalidate's multi_logloss: 0.0725497\n",
      "[64]\ttrain's multi_logloss: 0.105039\tvalidate's multi_logloss: 0.0709544\n",
      "[65]\ttrain's multi_logloss: 0.104078\tvalidate's multi_logloss: 0.0703405\n",
      "[66]\ttrain's multi_logloss: 0.103134\tvalidate's multi_logloss: 0.0701205\n",
      "[67]\ttrain's multi_logloss: 0.102229\tvalidate's multi_logloss: 0.0692772\n",
      "[68]\ttrain's multi_logloss: 0.101354\tvalidate's multi_logloss: 0.068559\n",
      "[69]\ttrain's multi_logloss: 0.100491\tvalidate's multi_logloss: 0.0673473\n",
      "[70]\ttrain's multi_logloss: 0.0995573\tvalidate's multi_logloss: 0.0674286\n",
      "[71]\ttrain's multi_logloss: 0.0986634\tvalidate's multi_logloss: 0.0674853\n",
      "[72]\ttrain's multi_logloss: 0.0978101\tvalidate's multi_logloss: 0.0672873\n",
      "[73]\ttrain's multi_logloss: 0.0969953\tvalidate's multi_logloss: 0.0673621\n",
      "[74]\ttrain's multi_logloss: 0.0962072\tvalidate's multi_logloss: 0.066834\n",
      "[75]\ttrain's multi_logloss: 0.0954358\tvalidate's multi_logloss: 0.06728\n",
      "[76]\ttrain's multi_logloss: 0.0946999\tvalidate's multi_logloss: 0.0666785\n",
      "[77]\ttrain's multi_logloss: 0.093984\tvalidate's multi_logloss: 0.0652261\n",
      "[78]\ttrain's multi_logloss: 0.093268\tvalidate's multi_logloss: 0.0653247\n",
      "[79]\ttrain's multi_logloss: 0.0925889\tvalidate's multi_logloss: 0.0654675\n",
      "[80]\ttrain's multi_logloss: 0.0919186\tvalidate's multi_logloss: 0.0649799\n",
      "[81]\ttrain's multi_logloss: 0.0912796\tvalidate's multi_logloss: 0.0638035\n",
      "[82]\ttrain's multi_logloss: 0.0906195\tvalidate's multi_logloss: 0.0638154\n",
      "[83]\ttrain's multi_logloss: 0.0899888\tvalidate's multi_logloss: 0.0642833\n",
      "[84]\ttrain's multi_logloss: 0.0893663\tvalidate's multi_logloss: 0.0636025\n",
      "[85]\ttrain's multi_logloss: 0.0887785\tvalidate's multi_logloss: 0.0626043\n",
      "[86]\ttrain's multi_logloss: 0.0881855\tvalidate's multi_logloss: 0.0623685\n",
      "[87]\ttrain's multi_logloss: 0.0875885\tvalidate's multi_logloss: 0.0627226\n",
      "[88]\ttrain's multi_logloss: 0.0870171\tvalidate's multi_logloss: 0.0624081\n",
      "[89]\ttrain's multi_logloss: 0.0864743\tvalidate's multi_logloss: 0.0625911\n",
      "[90]\ttrain's multi_logloss: 0.0859347\tvalidate's multi_logloss: 0.0620309\n",
      "[91]\ttrain's multi_logloss: 0.0854179\tvalidate's multi_logloss: 0.0622157\n",
      "[92]\ttrain's multi_logloss: 0.0849131\tvalidate's multi_logloss: 0.0617822\n",
      "[93]\ttrain's multi_logloss: 0.0844192\tvalidate's multi_logloss: 0.0619947\n",
      "[94]\ttrain's multi_logloss: 0.0839399\tvalidate's multi_logloss: 0.0614539\n",
      "[95]\ttrain's multi_logloss: 0.0834681\tvalidate's multi_logloss: 0.0616655\n",
      "[96]\ttrain's multi_logloss: 0.0830149\tvalidate's multi_logloss: 0.06134\n",
      "[97]\ttrain's multi_logloss: 0.0825657\tvalidate's multi_logloss: 0.0613612\n",
      "[98]\ttrain's multi_logloss: 0.0821295\tvalidate's multi_logloss: 0.0611025\n",
      "[99]\ttrain's multi_logloss: 0.0816869\tvalidate's multi_logloss: 0.0613398\n",
      "[100]\ttrain's multi_logloss: 0.0812595\tvalidate's multi_logloss: 0.0610704\n",
      "0.9777777777777777\n",
      "训练集 0.9717813051146384\n",
      "验证集 0.9734471313418682\n"
     ]
    }
   ],
   "source": [
    "import lightgbm as lgb\n",
    "from sklearn import datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "from sklearn.metrics import roc_auc_score, accuracy_score\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# 加载数据\n",
    "iris = datasets.load_iris()\n",
    "# 划分训练集和测试集\n",
    "X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3)\n",
    "# 转换为Dataset数据格式\n",
    "train_data = lgb.Dataset(X_train, label=y_train)\n",
    "validation_data = lgb.Dataset(X_test, label=y_test)\n",
    "# 参数\n",
    "results = {}\n",
    "params = {\n",
    "    'learning_rate': 0.1,\n",
    "    'lambda_l1': 0.1,\n",
    "    'lambda_l2': 0.9,\n",
    "    'max_depth': 1,\n",
    "    'objective': 'multiclass',  # 目标函数\n",
    "    'num_class': 3,\n",
    "    'verbose': -1 \n",
    "}\n",
    "# 模型训练\n",
    "gbm = lgb.train(params, train_data, valid_sets=(validation_data,train_data),valid_names=('validate','train'),evals_result= results)\n",
    "# 模型预测\n",
    "y_pred_test = gbm.predict(X_test)\n",
    "y_pred_data = gbm.predict(X_train)\n",
    "y_pred_data = [list(x).index(max(x)) for x in y_pred_data]\n",
    "y_pred_test = [list(x).index(max(x)) for x in y_pred_test]\n",
    "# 模型评估\n",
    "print(accuracy_score(y_test, y_pred_test))\n",
    "print('训练集',f1_score(y_train, y_pred_data,average='macro'))\n",
    "print('验证集',f1_score(y_test, y_pred_test,average='macro'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA4/0lEQVR4nO3deXxU9dX48c+Zyb4HErZESBAIAiK7qGjjWnApPLYqVK21tTxq1dbWp9qnv9ba2j62tdZuanFvrVDqrkVtaYnYKgoosqMISMIe1oSsM3N+f9wbZogJmQkzmSzn/XpdZ+693/u9534x98z93k1UFWOMMaYtnngHYIwxpmuwhGGMMSYsljCMMcaExRKGMcaYsFjCMMYYExZLGMYYY8JiCcN0WSLyvyLySBTq+aGIPBXvOGJFRF4VkWuiXdb0PGL3YZhoEpEtwABggKpWhkxfAZwCFKvqljbqKAWeUtXCWMXZbH0/BIao6lUdsb5IiIgCQ1V1Y7xjMcaOMEwsbAZmNY2IyMlAajRXICIJ0ayvveIdR7zXb3oWSxgmFv4EfClk/Brgj6EFRCRZRO4Vka0isktEHhKRVBFJB14FBohItTsMcLuNnhGRp0TkEPDl5l1JIjJFRN4SkQMiUi4iX24pOBEpFpE3RKRKRP4B5IXMKxWRimblt4jIee73Y8YhIkUioiJyjbttlSLyvZC6UkXkSRHZLyLrROQ7zdcXUnax+/UDtx2uaIpPRG4XkZ3A4yKSKyKviMget95XRKQwpJ4yEbnO/f5lEfm32/b7RWSziExrZ9liEVnstuNCEfn98XTtmc7PEoaJhSVAloicJCJe4Aqg+Y7kZ8AwYAwwBCgAfqCqh4FpwHZVzXCH7e4y04FngBzgz6GVichAnETzWyDfrXdFK/E9DSzHSRQ/xklokWg1jhBTgBLgXOAHInKSO/1OoAgYDJwPtNoNpqpnuV9PcdvhL+54P6AXMAiYjfN3/Lg7PhCoBX53jPhPBTbgbP/PgUdFRNpR9mngXaA38EPg6mOs03QDljBMrDQdZZwPrAe2Nc1wdzhfA25V1X2qWgX8FJjZRp1vq+oLqhpQ1dpm864EFqrqXFVtVNW9qrqieQVuYpkIfF9V61V1MfByhNt2rDia3KWqtar6AfABzvkbgMuBn6rqflWtAH4T4boBAsCdbvy17rY+q6o1blv+BPjMMZb/RFUfVlU/8CTQH+gbSdmQdvyBqjao6r+Bl9qxLaYLsf5PEyt/AhYDxTTrjsI5AkgDlof8sBXA20ad5ceYdwLwcRhxDQD2u0cyTT5xlw/XseJosjPkew2QEbL+0OXDqau5Papa1zQiImnAr4CpQK47OVNEvO6OvtXYVLXG/TfIaKHcscrmAftUtabZtkTSjqaLsSMMExOq+gnOye8Lgeeaza7E6TYZqao57pCtqk07rdYu3TvWJX3lwIlhhLYDyHXPlTQZGPL9ME4yA8DtUsuPII5w1h969Vd7drDN1/9tnO6vU1U1C2jqymqtmykadgC93GTVxJJFN2cJw8TSV4Fzmv2aR1UDwMPAr0SkD4CIFIjIZ90iu4DeIpIdwbr+DJwnIpeLSIKI9BaRMc0LuYlsGXCXiCSJyBTgkpAiHwIpInKRiCQC/w9IjiCOtswHvuueqC4Abmqj/C6c8x3HkomTgA+ISC+c8yQxFdKOP3Tb8TSObkfTDVnCMDGjqh+r6rJWZt8ObASWuFcbLcT5lYyqrgfmApvcK54GhLGurThHM98G9uGc8D6lleJfxDmZuw9n53qky0xVDwI3Ao/gnHc5DLR4FVM7/citbzPONj8D1B+j/A+BJ912uLyVMvfjXLZciXPBwWvRCrYNVwKnAXuBu4G/cOxtMV2c3bhnTByJyA3ATFU91knqLkFE/gKsV9WYH+GY+LAjDGM6kIj0F5EzRMQjIiU4R0TPxzuu9hCRiSJyorstU3EuN34hzmGZGLKrpIzpWEnAH3CuHjsAzAMeiGdAx6EfzgUNvXG62W5Q1ffjG5KJJeuSMsYYExbrkjLGGBOWLtkllZOTo0OGDIl3GJ3C4cOHSU9Pb7tgD2BtEWRtEWRtEbR8+fJKVW1+X1HYumTC6Nu3L8uWtXa1Zs9SVlZGaWlpvMPoFKwtgqwtgqwtgkTkk+NZ3rqkjDHGhMUShjHGmLBYwjDGGBOWLnkOwxjT8zQ2NlJRUUFdXV3bhUNkZ2ezbt26GEXVOaWkpFBYWEhiYmJU67WEYYzpEioqKsjMzKSoqIjW3/f0aVVVVWRmZsYwss5FVdm7dy8VFRUUFxdHtW7rkjLGdAl1dXX07t07omTRE4kIvXv3jvhILByWMIwxXYYli/DEqp0sYRhjjAmLJQxjjAnDgQMHeOCByJ8TeeGFF3LgwIHoBxQHljCMMSYMrSUMv7+l16YHLViwgJycnBhF1bHsKiljjAnDHXfcwccff8yYMWNITEwkIyOD/v37s2LFCtauXcuMGTMoLy+nrq6Ob3zjG8yePRuAoqIili1bRnV1NdOmTWPKlCm89dZbFBQU8OKLL5KamhrnLQufJQxjTJdz18trWLv9UFhl/X4/Xq+3zXIjBmRx5yUjW51/zz33sHr1alasWEFZWRkXXXQRq1evPnLp6mOPPUavXr2ora1l4sSJfP7zn6d3795H1fHRRx8xd+5cHn74YS6//HKeffZZrrrqqrC2ozOwhGGMMe0wadKko+5z+M1vfsPzzzsvTywvL+ejjz76VMIoLi5mzJgxAIwfP54tW7Z0VLhRYQnDGNPlHOtIoLlY3bgX+sj0srIyFi5cyNtvv01aWhqlpaUt3geRnJx85LvX66W2tjbqccWSnfQ2xpgwZGZmUlVV1eK8gwcPkpubS1paGuvXr2fJkiUdHF3HsCMMY4wJQ+/evTnjjDMYNWoUqamp9O3b98i8qVOn8tBDDzF69GhKSkqYPHlyHCONHUsYxhgTpqeffrrF6cnJybz66qstzms6T5GXl8fq1auPTL/tttuiHl+sWZeUMcaYsFjCMMYYExZLGMYYY8JiCcMYY0xYLGEYY4wJiyUMY4wxYbGEYYwxMZCRkQHA9u3b+cIXvtBimdLSUpYtW3bMeu6//35qamqiHl97WMIwxpgYGjBgAM8880y7l+8xCUNEHhOR3SKyupX5IiK/EZGNIrJSRMbFMh5jjGmv22+//aj3Yfzwhz/krrvu4txzz2XcuHGcfPLJvPjii59absuWLYwaNQqA2tpaZs6cyejRo7niiiuOepbUDTfcwIQJExg5ciR33nkn4DzQcPv27Zx99tmcffbZAPz973/ntNNOY9y4cVx22WVUV1fHcrOPEus7vZ8Afgf8sZX504Ch7nAq8KD7aYwxrXv1Dti5KqyiqX4feMPY1fU7Gabd0+rsmTNn8s1vfpMbb7wRgPnz5/Paa69x6623kpWVRWVlJZMnT+Zzn/tcq+/UfvDBB0lLS2PlypWsXLmSceOCv5F/8pOf0KtXL/x+P+eeey4rV67klltu4b777mPRokXk5eVRWVnJ3XffzcKFC0lPT+dnP/sZ9913Hz/4wQ/CaovjFdOEoaqLRaToGEWmA39UVQWWiEiOiPRX1R2xjMsYYyI1duxYdu/ezfbt29mzZw+5ubn079+fW2+9lcWLF+PxeNi2bRu7du2iX79+LdaxePFibrnlFgBGjx7N6NGjj8ybP38+c+bMwefzsWPHDtauXXvUfIAlS5awdu1azjjjDAAaGho47bTTYrTFnxbvZ0kVAOUh4xXutE8lDBGZDcwGGNgnh7Kyso6Ir9Orrq62tnBZWwR1x7bIzs4OPi12yvfCXi7cFygB0MrTaJtccsklPPXUU+zevZsZM2bw6KOPsmPHDsrKykhMTGTUqFFUVlYeefR5VVUV1dXVBAIBqqqq8Pl81NbWHtmOQCDA4cOHWbVqFT//+c8pKysjNzeX66+/ngMHDlBVVYWqUl1dTXJyMjU1NZSWlvL44483C/vTcdfV1UX9/4F4J4yWjtu0pYKqOgeYAzC+MFlLS0tjGFbXUVZWhrWFw9oiqDu2xbp169r1Xotovg/jmmuu4Wtf+xqVlZW88cYbzJ8/nwEDBtCrVy8WLVrE1q1bycjIOLK+zMxMMjIy8Hg8ZGZmcs455/D8888feVvf6tWrSU9PJxAIkJmZSWFhIXv27GHhwoWcf/75ZGZmkpWVhaqSmZnJ2WefzW233cauXbsYMmQINTU1VFRUMGzYsE/FmpKSwtixY6Oy3U3inTAqgBNCxguB7W0tJAEfHK6E9LyYBWaMMc2NHDmSqqoqCgoK6N+/P1deeSWXXHIJEyZMYMyYMQwfPvyYy99www1ce+21jB49mjFjxjBp0iQATjnlFMaOHcvIkSMZPHjwkS4ngNmzZzNt2jT69+/PokWLeOKJJ5g1axb19fUA3H333S0mjFiId8J4CbhJRObhnOw+GPb5ix0rYMh5MQzNGGM+bdWq4Mn2vLw83n777RbLNV29VFRUdOSx5qmpqcybN6/F8k888USL02+++WZuvvnmI+PnnHMOS5cubU/oxy2mCUNE5gKlQJ6IVAB3AokAqvoQsAC4ENgI1ADXhl35jg8sYRhjTAeK9VVSs9qYr8DXI623gUTYvqK9YRljjGmHLnmndy3J+C1hGNPjOL8xTVti1U5dM2FoEt6DW6FmX7xDMcZ0kJSUFPbu3WtJow2qyt69e0lJSYl63fE+6d0utSQDDbBzJQwujXc4xpgOUFhYSEVFBXv27Iloubq6upjsPDuzlJQUCgsLo15vl0wYjZIMVDnnMSxhGNMjJCYmUlxcHPFyZWVlUb8foafqkl1SCV4vOyTfuVLKGGNMh+iSCSPJCyt8Rfi3vR/vUIwxpsfooglDWB0oxntgM9QdjHc4xhjTI3TJhJHshdXq9mXuWBnfYIwxpofokgnDI1CZUeKM2HkMY4zpEF0yYQAMKBzEbuntPFPKGGNMzHXZhDFqQDYf+IoI2B3fxhjTIbpuwijIYlWgGNm7EeqP/dITY4wxx68LJ4xsVmkxgsJ2u7zWGGNircsmjD6ZyWxOHUUAD2z5d7zDMcaYbq/LJgwRYVDBADZ6B8PmN+MdjjHGdHtdNmGAcx7jjYbhaMVSaKiJdzjGGNOtde2EMSCb//hPQgKNUP5OvMMxxphurUsnjLEDc1kaGE5AvHYewxhjYqxLJ4x+2Snk5PTik+RhsMXOYxhjTCx16YQBMKEo1zmPsW051FfHOxxjjOm2un7CGJTLP+tKkIAPypfEOxxjjOm2unzCGD+oF8sCwwhIgl1ea4wxMdTlE0ZJv0wSkjPYmnqSnccwxpgYinnCEJGpIrJBRDaKyB0tzM8WkZdF5AMRWSMi10ZSv9cjjBmYw1v+Ec47vusORS12Y4wxQTFNGCLiBX4PTANGALNEZESzYl8H1qrqKUAp8EsRSYpkPRMG9eKVqiGgftj6dhQiN8YY01ysjzAmARtVdZOqNgDzgOnNyiiQKSICZAD7AF8kK5lQlMvywFACniTYvDgacRtjjGkmIcb1FwDlIeMVwKnNyvwOeAnYDmQCV6hqoHlFIjIbmA2Qn59PWVnZkXm1PqWBJD5KHEbByldYlnx+VDeiM6uurj6qLXoya4sga4sga4voiXXCkBamabPxzwIrgHOAE4F/iMibqnrUyQhVnQPMASgpKdHS0tKjKhmx9k2WN06ipOoxSscPh8x+0dmCTq6srIzmbdFTWVsEWVsEWVtET6y7pCqAE0LGC3GOJEJdCzynjo3AZmB4pCuaMCiXZw4Mc0Y2lbUnVmOMMccQ64SxFBgqIsXuieyZON1PobYC5wKISF+gBNgU6YrGF/Xi/cZCfCm94eNFxxm2McaY5mLaJaWqPhG5CXgd8AKPqeoaEbnenf8Q8GPgCRFZhdOFdbuqVka6rgmDclE8fJIzkRM//heogrTUI2aMMaY9Yn0OA1VdACxoNu2hkO/bgQuOdz0DclIpzE3lTf/JnHj4Ndi1BvqNOt5qjTHGuLr8nd6hzhyax1O7Bzsjm6xbyhhjoqlbJYwpQ/LZWJ9NbfYQ+Phf8Q7HGGO6lW6VMM4Y0hsRWJc2AT55Cxrr4h2SMcZ0G90qYeSkJTG6IJtXa08CX509JsQYY6KoWyUMgDOH5jNv90DUk2jdUsYYE0XdLmFMGZpHVSCZ/b3HWsIwxpgo6nYJY9zAXNKSvCxJnAS7VkPlxniHZIwx3UK3SxhJCR4mD+7No/vHAQKr5sc7JGOM6Ra6XcIA536M5ftTqDthCqz8i3PXtzHGmOPSbRMGwIqc82H/FqhYFt+AjDGmG+iWCePE/Az6Z6fwl+ox4E22biljjImCbpkwRIQzh+axcHMdgWFTYfVz4G+Md1jGGNOlhZ0wROQyEcl0v/8/EXlORMbFLrTjc95Jfamq87E+fyrUVNo7Mowx5jhFcoTxfVWtEpEpOG/JexJ4MDZhHb8zh+aTkujhrweHQ0o2rLRuKWOMOR6RJAy/+3kR8KCqvggkRT+k6EhN8nLW0HxeXbcfHTED1r8C9dXxDssYY7qsSBLGNhH5A3A5sEBEkiNcvsNdMLIfOw/Vsan/RdBYA+tejndIxhjTZUWyw78c5815U1X1ANAL+J9YBBUt5w7vg0fg+b2DILcIVvw53iEZY0yXFUnC6A/8TVU/EpFS4DLg3VgEFS256UlMKu7F39ftgjFXwZY3Yd/meIdljDFdUiQJ41nALyJDgEeBYuDpmEQVRReM6MeHu6rZOnA6IPDB3HiHZIwxXVIkCSOgqj7gUuB+Vb0V56ijUzt/RF8AXiv3wolnw4qnIRCIc1TGGNP1RJIwGkVkFvAl4BV3WmL0Q4quE3qlMaJ/Fn9fswvGXAkHy2HzG/EOyxhjupxIEsa1wGnAT1R1s4gUA0/FJqzo+uzIfizfup89hec792TYyW9jjIlY2AlDVdcCtwGrRGQUUKGq98QssiiaOqofqrBg3X44+TLn8traA/EOyxhjupRIHg1SCnwE/B54APhQRM4KY7mpIrJBRDaKyB2t1S0iK0RkjYhEvb+opF8mw/tl8tz725xuKV8drH422qsxxphuLZIuqV8CF6jqZ1T1LJzHg/zqWAuIiBcnwUwDRgCzRGREszI5OAnoc6o6Eudy3ai7dFwBH5QfYFPiUOh7Mix91N6TYYwxEYgkYSSq6oamEVX9kLZPek8CNqrqJlVtAOYB05uV+SLwnKpudevdHUFMYZs+pgAReGHFdph8A+xeYw8kNMaYCESSMJaJyKNu91GpiDwMLG9jmQKgPGS8wp0WahiQKyJlIrJcRL4UQUxh65uVwhkn5vH8im3oqM9Deh9Y8kAsVmWMMd1SQgRlbwC+DtwCCLAYpyvpWKSFac37gRKA8cC5QCrwtogscY9gghWJzAZmA+Tn51NWVhZB6I6TUhv5974GHn7lP5yffy7FH83l3b89RU16YcR1dRbV1dXtaovuyNoiyNoiyNoiesJOGKpaD9znDuGqAE4IGS8EtrdQplJVDwOHRWQxcApwVMJQ1TnAHICSkhItLS2NIAzHxHofT61fyBb6UPyFH8OvnmOSLofSqyKuq7MoKyujPW3RHVlbBFlbBFlbRE+bXVIiskpEVrY2tLH4UmCoiBSLSBIwE3ipWZkXgTNFJEFE0oBTgXXt2Zi2pCcn8NmRffnbyh3Up/SC0ZfDirlQsy8WqzPGmG4lnCOMi9tbuar6ROQmnKfceoHHVHWNiFzvzn9IVdeJyGvASiAAPKKqq9u7zrbMGFvACyu2s2j9bqZOvhHe/xMsewzOui1WqzTGmG6hzYShqp8czwpUdQGwoNm0h5qN/wL4xfGsJ1xThuSRn5nM/GUVTP3yRBh8Nrz7MJx+MyQkd0QIxhjTJUVy416ViBxqNpSLyPMiMjiWQUZTgtfD5RMKWbRhNxX7a+CMb0D1TntciDHGtCGSy2rvw3lhUgHOyevbgIdx7q14LPqhxc6sSQMB+MvSchhcCgXj4d+/An9jfAMzxphOLJKEMVVV/6CqVap6yL1q6UJV/QuQG6P4YqIwN42zS/owb2k5jQGFs/4HDmyFVc/EOzRjjOm0InofhohcLiIed7g8ZF6Xe8bGlacOZE9VPQvX7oJhU6HvKPj3ffauDGOMaUUkCeNK4GpgtztcDVwlIqnATTGILaZKS/owIDuFP7+zFUTgzG9D5YewrvlVv8YYYyCyx5tvUtVLVDXPHS5R1Y2qWquq/45lkLHg9QizJg3k3xsr2Vx5GEZMh95DYfG99lBCY4xpQSRXSRW6V0TtFpFdIvKsiHTdZ2oAV0w8Aa9HmPvuVvB4YcqtsGuVHWUYY0wLIumSehznLu0BOFdKvexO67L6ZKVwwYi+zF9WTk2DD0ZfAX1Gwmv/Cw2H4x2eMcZ0KpEkjHxVfVxVfe7wBJAfo7g6zFenFHOgppH5S8vBmwAX/RIOVcDiDrmP0BhjuoxIEkaliFwlIl53uArYG6vAOsqEol5MGJTLw29uptEfgEGnwSlfhLd+B3s+bLsCY4zpISJJGF8BLgd2AjuAL7jTurwbSk9k24FaXlnpPkj3/B9BUhos+LadADfGGFckV0ltVdXPqWq+qvZR1RnH+5ypzuLskj6U9M3kwbKPCQQUMvLhnO/D5sX27m9jjHG1+fBBEfktx7gxT1VviWpEceDxCNeXDubWv3zAog27OfekvjDhK86TbP9xJwy/CBJT4x2mMcbEVThHGMtwXsXa2tAtXDx6AAU5qTz0xsfOBI8XPvtT5wS4vcrVGGPCerz5k+FUJCK/VdWbjz+k+Ej0evjamcX88OW1vLt5H5OKe0HRFCi5CN78FYz9ktNVZYwxPVQkJ73bckYU64qLKyYOJD8zmXv/vgFtOtl9/l3gq4Wy/4tvcMYYE2fRTBhdXmqSl5vPGcK7m/fx5keVzsS8oc75jOVPwJ4NcY3PGGPiyRJGMzMnDqQgJ/Xoo4zP3A5J6fD69+wyW2NMjxXNhCFRrCtukhI8fOO8oaysOMjra3Y5E9PzoPQO2PgPeOPn8Q3QGGPiJJoJ49dRrCuuLh1bwOD8dO77xwb8AfeIYvKNcMosKPspvG+vczXG9DxtJgwRud/9fFlEXmo+NJVzny3VLSR4Pdx63jA+3FXNSx9scyaKwCW/cV7p+vItsPGfcY3RGGM6WpuX1QJ/cj/vjWUgnc1FJ/fnoTc+5hevbeCzI/uRlpQACUlw+R/hsWkw/xr4yqvQ7+R4h2qMMR2izSMMVW26OW+Mqr4ROgBjYhpdHHk8wp2XjGT7wToeLPs4OCMlG678KyRnwtMzoWpX/II0xpgOFMk5jGtamPblKMXRKU0q7sX0MQP4w+JNbN1bE5yRXQCz5kLtPpg3Cxpr4xekMcZ0kHDOYcwSkZeB4mbnLxYRxuPNRWSqiGwQkY0icscxyk0UEb+IfCGyTYit7047iQSP8KNX1h49Y8AYuPRh2PYevHCjXW5rjOn2wjmH8RbO48zzgF+GTK8CVh5rQRHxAr8HzgcqgKUi8pKqrm2h3M+A18MPvWP0y07h5nOG8rPX1lO2YTelJX2CM0+6GM77ISy8E/KHQ+ntcYvTGGNiLZxnSX0CfAKc1o76JwEbVXUTgIjMA6YDzX6uczPwLDCxHeuIua9MKeKvy8q56+W1TB7cm5REb3DmGd+APeudR4cMPNW5isoYY7qhcB5vXkXLjzcXQFU16xiLFwDlIeMVwKnN6i8A/gs4h2MkDBGZDcwGyM/Pp6ysrK3Qo+rSIj/3Lqvjtsf/yReGJR01z5M1g/Fpb5I49xqWTbifhuTcDoururq6w9uis7K2CLK2CLK2iJ5wjjAyj6P+lu7+bp587gduV1W/SOs3i6vqHGAOQElJiZaWlh5HWJErBbboBzz//jZuuPhURg7IPrrAqIHw8DmcvvNxuPp55/HoHaCsrIyObovOytoiyNoiyNoiesK+SkpEBrY0tLFYBXBCyHghsL1ZmQnAPBHZgvPa1wdEZEa4cXWk7180gl7pSXznmZX4/IGjZ/YdARf+HDa/AW/+suUKjDGmC4vkstq/hQz/BDYBr7axzFJgqIgUi0gSMBN4KbSAqharapGqFgHPADeq6gsRxNVhstMS+fH0kazZfog5b276dIGxV8PJl8Gin8B/fmNXThljupVwrpICQFWPuqVZRMYB/93GMj4RuQnn6icv8JiqrhGR6935D0UecnxNHdWfaaP6cf/Cj7hgRF+G9AnpsROBz/0O/I3wj+/DwXKYek+HdU8ZY0wshZ0wmlPV90SkzauaVHUBsKDZtBYThap+ub3xdKS7po/knc37uHnuCp6/8fSjr5pKTIEvPA4LT4C3fgsHK+Dzj0JSWvwCNsaYKIjkHMa3QobbRGQusCeGsXVafTJTuPey0azbcYj/W7Du0wU8Hrjgbpj2C/jwNXjqUqg72PGBGmNMFEVyDiMTyHCHJOBl4HOxCKorOGd4X66bUsyTb3/C62t2tlzo1NnO0UXFUnjyEjjc5o3xxhjTaUWSMBYAY3HumZgFfBfnpHaP9Z2pwzm5IJvvPLOSbQdaeZ7UqEth5tOwez08cSEc2tGxQRpjTJREkjCeAh4DLgUudodLYhFUV5GU4OG3s8biDyhf//N71DX6Wy447LNw1TPO+Yw5n4EPO90TUIwxpk2RJIw9qvqyqm5W1U+ahphF1kUU5aVz72WjWVF+gO+/sDr4HvDmis+Cr7wOaXnw9OXw8jegvrpjgzXGmOMQScK4U0QecZ9ee2nTELPIupCpo/pzyzlD+OvyCp58a0vrBfuNgtmLnOdPLX8SHjoD9mzosDiNMeZ4RJIwrsV5YdJUnK6oS3C6pQzwzfOGcd5Jffnx39bx1sbK1gsmJMP5P4JrF0BDDTx6AXzyVscFaowx7RRJwjhFVSeo6jWqeq07fCVmkXUxHo/wqytOoTgvnRuffo+P97TR3TTodLjuH5CeD3+cAWue75A4jTGmvSJJGEtEZETMIukGMlMSefSaCXhF+NKj77LrUN2xF8gtgq/+HQaMhb9eC3/7NlTv7pBYjTEmUpEkjCnACvfteStFZJWIHPMFSj3RoN7pPH7tRPbXNPDlx5dyqK7x2Auk9YIvvQATr4Nlj8Ovx8Ci/4P6qo4I1xhjwhZJwpgKDAUuIHj+okdfVtua0YU5PHTVeD7aVcXsPy6j3tfK5bZNElPhonvh6+/C0PPgjXvgt+Nh5V/tAYbGmE4j7IQReimtXVbbtrOG5XPvZaewZNM+bnzqvbaTBkDeELj8j3DdPyFrADx3HTxxMexu4fEjxhjTwSI5wjARmjG2gLtnjOKf63fz9T+/R4Mv0PZCAIUTnKRx8f2wew08eDrM/SJ8vMiOOIwxcWMJI8aumjyIH88YxcJ1u7kxkqTh8cKEa+Gm5TDlVih/B/40A343Ed74hXPUYcnDGNOBLGF0gKsnD+LH00eycN0ubnhqObUNYXRPNUnvDef+AL61Fv5rjnOSfNHd8MBk+O04Bm2ZZyfIjTEdwhJGB7n6tCLunjGKf23YzdWPvsPBmjaunmouIRlOucK5DPdb6+Gi+yBnEMVb5sJvxsGyx8Dvi03wxhiDJYwOddXkQfxu1jhWVhzksj+8xc6Dbdyn0Zqs/jDxq/ClF1g+7ufQ+0R45VZ44FT4192wdYklD2NM1FnC6GAXje7PE9dOZPuBOi594D+s3X7ouOqryiqBa1+FK55yHmz45i/hsc/CLwbDK9+Cyo1RitwY09NZwoiD04fkMW/2ZAIKn3/wLV5ddZzvyBCBky6Br74O39kElz0Jw6bB+3+C302AubNg7UuwcxXUHojKNhhjep52v9PbHJ9RBdm8dNMZzP7Tcm7483t887yh3HLOUDweOb6KU3Nh5AxnOP9HsPQRZ9gQ8lr11F4wYjqMvQoKxjsJx5iO4vdB/SGo3Q91B6DhMCSlQ0oOpGSDv9GZXnsA/A2QkuVMT8pwLvCoO+AsizjTk7MgOQO8yeBNBG+SM3ic38MS8EPVTuexO/VVzrqSMyExzXl18uHdzjxfvbt8IngSwZPgDl5QvxN3oNG5OrFpuscL4g2ON9Y629NQ7WxHUx3igYDP2Z6A213s8TrTpenT/f3eeNit4zAE/CHlQgdx6gn4nfU0VDvtVXcAGmuC9Xq8cOZtcMLEqPzTWcKIoz5ZKcybPZnvPb+a+xd+xKqKg9x72SnkpidFZwWZfeGc78GZ34Jda+HgVjhQDjtXwgfzYPnjkD8chpwH/U52hrxhzh+M6bkCfmdHWrvf+aw74DxZOeBzd3qN4KuFxjrw1Tk7wSNDY7CMvzG4fO2BkLo66D0w4gVvEmf56mBxN78EPSnDSbipuc6TIzTgJLmA3/k3ihJLGHGWkujl3stGM7owm7v/tpaLfvMmv/3iOMYPyo3eShJToXC8MzSpOwRrnoMP/uIcgTT9T+VJdE6i5w2DvKGQ1tv5BZeSBb1OdBKM1/636RCNdcEdd9Pgq3V+4Qb87o653vll7Ktzd5DOL+TC8rWw6C1nmfqq4K/RgM+tcx/U7HeWb/o1jTrz6ttxXs2T4P6qd39RN/1KT8mG1BznQZupOc5405FE03hShvNruimheBLceTlOnfVVzryGaqdsaq4zX9WJte4QNFQ5RwD+BmebQr5/sm03RaMmQXof58iiscaps+Gws/6MPs68xJTgUYS/IdheAZ/btgnONom489z5GvI9MdU5gknKcNrgSJmA82u/qY3AmRbwO8urOuPgHPkkpTuDxwuBQDABNJXTQLCdm444OoD95XcCIsI1pxcxdmAOX3/6Pa74w9t8+4ISZp81GO/xdlG1JiULxn/ZGfw+2LvROcexew3s+RB2rYH1rwT/J26SkAL9RkPfkZBd6AxZBe4fXb7zR+6J46mxQMDdYTSGdC14nT/axhqnywCcHUVCivOHpursPOoPuTu5LOcyZlWo2QfVO+Gw+46Tpi6BxppgF0DTTra+yvkl7k10dhwJKc6Op+6QM7/hcPCPPeBzd4SHoP6gE3fTDgl1pvvr290MQwA+FmdbkrNCulASnPGsAuh7srOdTTtFVXdHnhvcYR/ZqacHE4s3wdm2hBRnO73J8f03b8OWsjKKJpbGO4z283hwTjfHf3cd8whEZCrwa8ALPKKq9zSbfyVwuztaDdygqh/EOq7OaHRhDq/cfCbffW4lP3ttPQvX7eKXl51CUV56bFfsTYA+w52By4LTA/7gL7i6g1D5IWx/3xnWvuj8Sm1OvM5OpGnn5ElwdkoJqZDgdrU19Q6IuDt1t0xylvMLMDnT2SmrOoV9dc7OtelXYVP3h6/hqK6RsxpqoKyly4klZKUhPInOr7mGaufX21Ftkuws428Isw2TnbiT0pxk1VjrxO1NCh6hJaYF+7M9CZAzMNgH7/G63Tnu/Tkp2cEhNdcZUnKcHXdT0vJ43R13srN+dfuz/Q38e8lSppx7oVPGmCiJacIQES/we+B8oAJYKiIvqerakGKbgc+o6n4RmQbMAU6NZVydWXZqIr//4jheWLGNO19cw7Rfv8kd04Zz9eRBx39CPFIeb3BnBdB/NJz8heD8hho4tM0ZDle6wx5nZ9l0qH5kx17ndJ2Ae/gs7qF8U594PRze7P5SP+Ts3wXnP6HJpOlQ3ZvkJJuEVKcrITGN8u27GDR4mFPek3B0X7o3ye0uSHN+4R/51V/j1Nv0S9zf6Pzib7p7PrM/ZPZzLlkWj9stEHB2/qG/wBOSO+pfJSy+xExLFibqYn2EMQnYqKqbAERkHjAdOJIwVDX0/aRLgMIYx9TpiQj/NbaQ0wbncfuzK7nzpTW8sGIbP5lxMiMGZMU7vKCkNOc8R97QeEcCwOayMgaVlsY7DGO6LdEYPsBORL4ATFXV69zxq4FTVfWmVsrfBgxvKt9s3mxgNkB+fv74+fPnxyzuzkRVeWu7j3kbGjjcCBcMSmDGkCRSEpyjjerqajIyMuIcZedgbRFkbRFkbRF09tlnL1fVCe1dPtZHGC31obSYoUTkbOCrOG/2+/RCqnNwuqsoKSnR0h70S/Js4Os1DfzstfXMfbec5Xu9fPv8YVw24QTeXPwGPaktjqWsrMzawmVtEWRtET2xvrShAjghZLwQ2N68kIiMBh4Bpqvq3hjH1CXlpCXxf5eO5rkbT+eE3FTueG4VF/3mTVbt8RHLo0RjjGkS64SxFBgqIsUikgTMBF4KLSAiA4HngKtV9cMYx9PljRuYy7M3nM7vvjiWww0+frm8niv+sIR3N7dwxZIxxkRRTBOGqvqAm4DXgXXAfFVdIyLXi8j1brEfAL2BB0RkhYgsi2VM3YGIcPHoASz81me46qQkNu89zOV/eJurH32HdzbttSMOY0xMxPw+DFVdACxoNu2hkO/XAZ86yW3alpzg5bxBiXxv1pk8teQTHnrjY66Ys4Txg3K54TMncs7wPh1/Ka4xptuK/62D5rilJnn52lmDufq0QcxfVs4f3tjEdX9cxuD8dL58ehGfH1dIerL9Uxtjjk/nvZ/fRCwl0cuXTiui7H9K+fXMMWQmJ/CDF9cw+af/5K6X1/DhLnuVqzGm/exnZzeU6PUwfUwB08cU8N7W/Tzxny08teQTHv/PFsYNzGHmpIFceHJ/MuyowxgTAdtjdHPjBuYybmAue6tH8Pz725j77la+88xKfvDias47qS//NbaAM4fmk5RgB5vGmGOzhNFD9M5I5rozB/PVKcW8t/UAL67YxssfbOeVlTvITk3kghF9uXB0f844Mc+ShzGmRZYwehgRYfygXMYPyuX7F49g8Yd7+NvKHby2eid/XV5BZkoCZ5f04bwRfSktyScrxV6mZIxxWMLowRK9Hs49qS/nntSXep+fNz+s5PU1O/nX+t289MF2EjxOcvlMST6lw/pwUv9MxF7nakyPZQnDAO49HSP6ct6IvvgDyvtb97Nw3W7e+HAPP39tAz9/bQN5GUlMHtyb00/M47QTe1PUO80SiDE9iCUM8ylejzChqBcTinpxx7Th7DpUxxsf7uHtj/fyn42VvLJyBwD5mclMKurFxKJcxg/qxfD+mSR67fyHMd2VJQzTpr5ZKVw+4QQun3ACqsqmysO8/fFelm7Zx9LN+/jbKieBpCR6GF2Yw9gTchhdmMPowmwKc1PtKMSYbsIShomIiHBifgYn5mdw1eRBAFTsr+G9rQd475P9vL91P4//ZwsNfudd4LlpiYwYkMWI/lmMHJBNSb9MTszPsCuxjOmCLGGY41aYm0ZhbhqfO2UAAPU+Pxt2VrGy4iCrKg6ybuchnnz7Exp8ThJJ8AiD89MZ2jeToX0yGOIORb3TSUm014oa01lZwjBRl5zgdbukco5M8/kDfLznMOt3HmLDzio27KxiVcVBFqzaQdPDdUVgQHYqg/PTKc5LZ1DvdIp6pzGwl5OQUpMsmRgTT5YwTIdI8Hoo6ZdJSb/Mo6bXNvjZVFnNxt3VbK48zObKw2zac5jn399GVZ3vqLL5mcmckJtKYW4aBbmpFOSkMiAnhf7ZqQzITrXHuhsTY5YwTFylJnkZOSCbkQOyj5ququyvaWRz5WEq9tewdW8N5ftrKN9Xy4ryAyxYtQNf4OgEkeyFgvfK6JeVQr+sFPpkpdAnM5m+WSn0yUomPyOZ/Mxke3KvMe1kfzmmUxIReqUn0Ss9ifGDcj813x9QdlfVsf1AHTsO1rLjQB1L13xEYlYWOw7W8s7mfeyuqqPR/+mjjtREL3mZSeRlJLtDkruuZHqnJ5GbnkTv9CRy0hLplZ5EaqLXrvQyBksYpovyeoT+2an0z04FnIQyNLCV0tJxR8o0HaXsrqpj96F69lTVs7uqnspqZ9hb3UD5vhpWlB9g3+EG/IGWu7SSEjzkpiWSk5pEdloiOamJZIcOaYlkpSSSlZrgfjrjmSkJpCVZsjHdhyUM022FHqUM73fssoGAcrC2kX01Dew/3MA+d9hf08iBmgb21zRwsLaRAzWNfLK3hoO1jRysbaS20X/Mej0CGckJZLoJJCM5gfTkBDJSEshIcr8ne0l3p6cne0l3p6clOdNTE71HvicneCwBmbixhGEM4PEIuW53FPnhL9fgC3CwtpFDdY0ccpNIVZ3PHRqDn/XOtMP1PvbXOEc21fXO+OGGYyedUCKQluglNckZ0hITSEnykpboJSXRQ2qSl5QEL/sq61lctZbUJA8pCV5S3PnJie73BPe7+5mc4HGG0O8JXhK9YgnKHGEJw5jjkJTgIT/TOZneXoGAUtPop6be5yYRPzUNPmoa/FTX+6htcMcb/e53Z6ht8FHb6Ke2MUBtg4/Kane8wU9VjY9lu8upafDRSk9b2JITPCS5SSTJ6ySVJK8zLcmdltg0L8FDoldIDJmWFDrNe/T3JK+HhBbmNU1L8ATnJXiFRI/zGfq9qZzXY8kt1ixhGBNnHo+Qkex0V/WJUp1lZWWUlpaiqjT6lTqfn7pGP/WNAep9fuqafdY3Bqjz+WnwBaj3BahrdL43jdf7AjT4g+ONIeMNvgA1tY3udz++gB41v9GvNPgDrZ4jiqZEr5DgcRJIglfwejwEfA2kvfMvEr0evB45Mq+pnDekbGKz8SPzPYLH/fQe+fR8arrXI3glOO3IPAnO9zQb93rA6/G4y3FkXmg5jwTLekKmfbqu0BiCdUUrkVrCMKYbExGSEoSkBE/c320SCCiNASeBNPoCR333BQI0+JRGf/C7LxCg0e+U8fmbxhWfP0BjwPn0+Z06fc2nBxR/wEmWFdu2k9+nF76AU4dTl7OugDp11zUG8AX8+EPmBwLOp88fwK9Ofb6A4vcrftUj6+iIRHg85lw9ngtGtnESL0yWMIwxHcLjEZI9XpITgPb34EWsrGwvpaVjYla/qhJQ8AUCBAJHf/pVPzUtoIo/ZFpTMmoaAk3j6iSt4DSOmuZXRUOm+/0B/Ar+QAB/gCP1nNgnI2rbGvOEISJTgV8DXuARVb2n2Xxx518I1ABfVtX3Yh2XMcZEg4jgFfB6mh5d030fYRPTR4aKiBf4PTANGAHMEpERzYpNA4a6w2zgwVjGZIwxpn1i/YzpScBGVd2kqg3APGB6szLTgT+qYwmQIyL9YxyXMcaYCMW6S6oAKA8ZrwBODaNMAbAjtJCIzMY5AiE/P5+ysrJox9olVVdXW1u4rC2CrC2CrC2iJ9YJo6VruZpfUhBOGVR1DjAHoKSkREtLS487uO6g6fJJY20RytoiyNoiemLdJVUBnBAyXghsb0cZY4wxcRbrhLEUGCoixSKSBMwEXmpW5iXgS+KYDBxU1R3NKzLGGBNfMe2SUlWfiNwEvI5zrdljqrpGRK535z8ELMC5pHYjzmW118YyJmOMMe0T8/swVHUBTlIInfZQyHcFvh7rOIwxxhyfWHdJGWOM6SYsYRhjjAmLJQxjjDFhsYRhjDEmLJYwjDHGhMUShjHGmLBYwjDGGBMWSxjGGGPCYgnDGGNMWCxhGGOMCYslDGOMMWGxhGGMMSYsljCMMcaExRKGMcaYsFjCMMYYExZLGMYYY8JiCcMYY0xYLGEYY4wJiyUMY4wxYbGEYYwxJiyWMIwxxoRFVDXeMURMRKqADfGOo5PIAyrjHUQnYW0RZG0RZG0RVKKqme1dOCGakXSgDao6Id5BdAYisszawmFtEWRtEWRtESQiy45neeuSMsYYExZLGMYYY8LSVRPGnHgH0IlYWwRZWwRZWwRZWwQdV1t0yZPexhhjOl5XPcIwxhjTwSxhGGOMCUuXSxgiMlVENojIRhG5I97xdBQROUFEFonIOhFZIyLfcKf3EpF/iMhH7mduvGPtKCLiFZH3ReQVd7xHtoWI5IjIMyKy3v3/47Qe3Ba3un8fq0Vkroik9JS2EJHHRGS3iKwOmdbqtovId9396AYR+Ww46+hSCUNEvMDvgWnACGCWiIyIb1Qdxgd8W1VPAiYDX3e3/Q7gn6o6FPinO95TfANYFzLeU9vi18BrqjocOAWnTXpcW4hIAXALMEFVRwFeYCY9py2eAKY2m9bitrv7jpnASHeZB9z96zF1qYQBTAI2quomVW0A5gHT4xxTh1DVHar6nvu9CmenUICz/U+6xZ4EZsQlwA4mIoXARcAjIZN7XFuISBZwFvAogKo2qOoBemBbuBKAVBFJANKA7fSQtlDVxcC+ZpNb2/bpwDxVrVfVzcBGnP3rMXW1hFEAlIeMV7jTehQRKQLGAu8AfVV1BzhJBegTx9A60v3Ad4BAyLSe2BaDgT3A42733CMikk4PbAtV3QbcC2wFdgAHVfXv9MC2CNHatrdrX9rVEoa0MK1HXRcsIhnAs8A3VfVQvOOJBxG5GNitqsvjHUsnkACMAx5U1bHAYbpvl8sxuf3z04FiYACQLiJXxTeqTqtd+9KuljAqgBNCxgtxDjl7BBFJxEkWf1bV59zJu0Skvzu/P7A7XvF1oDOAz4nIFpxuyXNE5Cl6ZltUABWq+o47/gxOAumJbXEesFlV96hqI/AccDo9sy2atLbt7dqXdrWEsRQYKiLFIpKEc9LmpTjH1CFERHD6qdep6n0hs14CrnG/XwO82NGxdTRV/a6qFqpqEc7/A/9S1avomW2xEygXkRJ30rnAWnpgW+B0RU0WkTT37+VcnHN9PbEtmrS27S8BM0UkWUSKgaHAu21V1uXu9BaRC3H6r73AY6r6k/hG1DFEZArwJrCKYL/9/+Kcx5gPDMT5g7lMVZuf+Oq2RKQUuE1VLxaR3vTAthCRMTgn/5OATcC1OD8Ge2Jb3AVcgXNV4fvAdUAGPaAtRGQuUIrzOPddwJ3AC7Sy7SLyPeArOG31TVV9tc11dLWEYYwxJj66WpeUMcaYOLGEYYwxJiyWMIwxxoTFEoYxxpiwWMIwxhgTFksYpkcSkWr3s0hEvhjluv+32fhb0azfmHixhGF6uiIgooQRxlM9j0oYqnp6hDEZ0ylZwjA93T3AmSKywn2XgldEfiEiS0VkpYj8Nzg3CLrvI3ka5+ZJROQFEVnuvn9htjvtHpynpa4QkT+705qOZsSte7WIrBKRK0LqLgt5p8Wf3TuVEZF7RGStG8u9Hd46xoRIiHcAxsTZHbh3igO4O/6DqjpRRJKB/4jI392yk4BR7uOgAb6iqvtEJBVYKiLPquodInKTqo5pYV2XAmNw3lmR5y6z2J03FufdBNuB/wBniMha4L+A4aqqIpIT3U03JjJ2hGHM0S4AviQiK3Aeu9Ib5zk7AO+GJAuAW0TkA2AJzoPchnJsU4C5qupX1V3AG8DEkLorVDUArMDpKjsE1AGPiMilQM1xbpsxx8UShjFHE+BmVR3jDsXuOxXAeXS4U8h5htV5wGmqegrOc4tSwqi7NfUh3/1Agqr6cI5qnsV58c1rEWyHMVFnCcP0dFVAZsj468AN7qPkEZFh7guJmssG9qtqjYgMx3ltbpPGpuWbWQxc4Z4nycd5U16rTwh1332SraoLgG/idGcZEzd2DsP0dCsBn9u19ATO+7GLgPfcE897aPmVnq8B14vISmADTrdUkznAShF5T1WvDJn+PHAa8AHOy2q+o6o73YTTkkzgRRFJwTk6ubVdW2hMlNjTao0xxoTFuqSMMcaExRKGMcaYsFjCMMYYExZLGMYYY8JiCcMYY0xYLGEYY4wJiyUMY4wxYfn/94u2+Rx9s/0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 有以下曲线可知验证集的损失是比训练集的损失要高，所以模型可以判断模型出现了过拟合\n",
    "lgb.plot_metric(results)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA4GUlEQVR4nO3deZwU1bnw8d/Ty+z7MMDAgAy7gOyyiMvgFhCXxLhg1Kj3TbiaGI3vNdHc5Ea9V3PNqzFmcQkuUeOCBnfjroy4oYICsoggIAz7Oguz9zzvH1XDNMMs3cP09PTM8/186tNd1aeqnzpKP3POqTolqooxxhjTGk+0AzDGGBMbLGEYY4wJiSUMY4wxIbGEYYwxJiSWMIwxxoTEEoYxxpiQWMIwMUtE/lNEHmyH49wsIo9HO45IEZHXROSy9i5ruh+x+zBMexKRjUAfoI+q7g7avhQYA+Sr6sZWjlEAPK6qeZGKs9H33QwMVtVLOuL7wiEiCgxR1XXRjsUYa2GYSNgAXFS/IiLHAInt+QUi4mvP47VVtOOI9veb7sUShomEfwA/DFq/DHgsuICIxIvInSKySUR2iMj9IpIoIsnAa0AfESlzlz5ut9F8EXlcREqAyxt3JYnI8SLykYjsF5HNInJ5U8GJSL6IvCcipSLyFtAj6LMCESlqVH6jiJzqvm8xDhEZICIqIpe557ZbRH4ddKxEEXlURPaJyGoR+WXj7wsqu9B9u8ythwvr4xORG0RkO/B3EckUkVdEZJd73FdEJC/oOIUi8iP3/eUi8oFb9/tEZIOIzGxj2XwRWejW49sics+RdO2Zzs8ShomERUCaiBwtIl7gQqDxD8nvgaHAWGAw0Bf4raoeAGYCW1U1xV22uvucA8wHMoAngg8mIv1xEs1fgBz3uEubie9JYAlOovgfnIQWjmbjCHI8MAw4BfitiBztbr8JGAAMBE4Dmu0GU9UT3bdj3Hp42l3vDWQBRwFzcP4d/91d7w9UAH9tIf7JwBqc8/9/wEMiIm0o+yTwKZAN3Axc2sJ3mi7AEoaJlPpWxmnAV8CW+g/cH5wfA9ep6l5VLQV+B8xu5Zgfq+oLqlqnqhWNPrsYeFtVn1LVGlXdo6pLGx/ATSzHAv+lqlWquhB4OcxzaymOereoaoWqLgOW4YzfAFwA/E5V96lqEfDnML8boA64yY2/wj3XZ1W13K3L24CTWtj/W1V9QFUDwKNALtArnLJB9fhbVa1W1Q+Al9pwLiaGWP+niZR/AAuBfBp1R+G0AJKAJUF/2ArgbeWYm1v4rB/wTQhx9QH2uS2Zet+6+4eqpTjqbQ96Xw6kBH1/8P6hHKuxXapaWb8iIknAH4EZQKa7OVVEvO4PfbOxqWq5+98gpYlyLZXtAexV1fJG5xJOPZoYYy0MExGq+i3O4PcZwHONPt6N020yUlUz3CVdVet/tJq7dK+lS/o2A4NCCG0bkOmOldTrH/T+AE4yA8DtUssJI45Qvj/46q+2/MA2/v7/wOn+mqyqaUB9V1Zz3UztYRuQ5SarepYsujhLGCaS/g9wcqO/5lHVOuAB4I8i0hNARPqKyHfcIjuAbBFJD+O7ngBOFZELRMQnItkiMrZxITeRLQZuEZE4ETkeOCuoyNdAgojMEhE/8BsgPow4WvMM8Ct3oLovcHUr5XfgjHe0JBUnAe8XkSyccZKICqrHm916nMqh9Wi6IEsYJmJU9RtVXdzMxzcA64BF7tVGb+P8lYyqfgU8Bax3r3jqE8J3bcJpzfwHsBdnwHtMM8V/gDOYuxfnx/Vgl5mqFgM/AR7EGXc5ADR5FVMb/bd7vA045zwfqGqh/M3Ao249XNBMmbtxLlvejXPBwevtFWwrLgamAnuAW4GnaflcTIyzG/eMiSIRuQqYraotDVLHBBF5GvhKVSPewjHRYS0MYzqQiOSKyDQR8YjIMJwW0fPRjqstRORYERnknssMnMuNX4hyWCaC7CopYzpWHPA3nKvH9gPzgHujGdAR6I1zQUM2TjfbVar6RXRDMpFkXVLGGGNCYl1SxhhjQhKTXVIZGRk6ePDgaIfRKRw4cIDk5OTWC3YDVhcNrC4aWF00WLJkyW5VbXxfUchiMmH06tWLxYubu1qzeyksLKSgoCDaYXQKVhcNrC4aWF00EJFvj2R/65IyxhgTEksYxhhjQmIJwxhjTEhicgzDGNP91NTUUFRURGVlZeuFg6Snp7N69eoIRdU5JSQkkJeXh9/vb9fjWsIwxsSEoqIiUlNTGTBgAM0/7+lwpaWlpKamRjCyzkVV2bNnD0VFReTn57frsa1LyhgTEyorK8nOzg4rWXRHIkJ2dnbYLbFQWMIwxsQMSxahiVQ9WcIwxhgTEksYxhgTgv3793PvveHPE3nGGWewf//+9g8oCixhGGNMCJpLGIFAU49Nb/Dqq6+SkZERoag6ll0lZYwxIbjxxhv55ptvGDt2LH6/n5SUFHJzc1m6dCmrVq3iu9/9Lps3b6ayspJrr72WOXPmADBgwAAWL15MWVkZM2fO5Pjjj+ejjz6ib9++vPjiiyQmJkb5zEJnCcMYE3NueXklq7aWhFQ2EAjg9XpbLTeiTxo3nTWy2c9vv/12VqxYwdKlSyksLGTWrFmsWLHi4KWrDz/8MFlZWVRUVHDsscfy/e9/n+zs7EOOsXbtWp566ikeeOABLrjgAp599lkuueSSkM6jM7CEYYwxbTBp0qRD7nP485//zPPPOw9P3Lx5M2vXrj0sYeTn5zN27FgAJkyYwMaNGzsq3HZhCcMYE3Naagk0Fqkb94KnTC8sLOTtt9/m448/JikpiYKCgibvg4iPjz/43uv1UlFR0e5xRZINehtjTAhSU1MpLS1t8rPi4mIyMzNJSkriq6++YtGiRR0cXcewFoYxxoQgOzubadOmMWrUKBITE+nVq9fBz2bMmMH999/P6NGjGTZsGFOmTIlipJFjCcMYY0L05JNPNrk9Pj6e1157rcnP6scpevTowYoVKw5uv/7669s9vkizLiljjDEhsYRhjDEmJJYwjDHGhMQShjHGmJBYwjDGGBMSSxjGGGNCYgnDGGMiICUlBYCtW7dy3nnnNVmmoKCAxYsXt3icu+++m/Ly8naPry0sYRhjTAT16dOH+fPnt3l/SxjGGBNjbrjhhkOeh3HzzTdzyy23cMoppzB+/HiOOeYYXnzxxcP227hxI6NGjQKgoqKC2bNnM3r0aC688MJD5pK66qqrmDhxIiNHjuSmm24CnAkNt27dyvTp05k+fToAb775JlOnTmX8+PGcf/75lJWVRfK0D2F3ehtjYs9rN8L2L0MqmhioBW8IP3W9j4GZtzf78ezZs/n5z3/OT37yEwCeeeYZXn/9da677jrS0tLYvXs3U6ZM4eyzz272mdr33XcfSUlJLF++nOXLlzN+/PiDn912221kZWURCAQ45ZRTWL58Oddccw133XUXCxYsoEePHuzevZtbb72Vt99+m+TkZH7/+99z11138dvf/jakujhSljCMMSYE48aNY+fOnWzdupVdu3aRmZlJbm4u1113HQsXLsTj8bBlyxZ27NhB7969mzzGwoULueaaawAYPXo0o0ePPvjZM888w9y5c6mtrWXbtm2sWrXqkM8BFi1axKpVq5g2bRoA1dXVTJ06NUJnfLiIJgwReRg4E9ipqqOa+FyAPwFnAOXA5ar6eWvH9dV2jv48Y0yUtNASaKyiHac3P++885g/fz7bt29n9uzZPPHEE+zatYslS5bg9/sZMGBAk9OaB2uq9bFhwwbuvPNOPvvsMzIzM7n88subPI6qctppp/HUU0+1y/mEK9JjGI8AM1r4fCYwxF3mAPeFctCEqp1HHJgxxoRr9uzZzJs3j/nz53PeeedRXFxMz5498fv9LFiwgG+//bbF/U888USeeOIJAFasWMHy5csBKCkpITk5mfT0dHbs2HHIRIbB06pPmTKFDz/8kHXr1gFQXl7O119/HYlTbVJEWxiqulBEBrRQ5BzgMVVVYJGIZIhIrqpua+m4UlcLJVshrU97hmuMMS0aOXIkpaWl9O3bl9zcXC6++GLOOussJk6cyNixYxk+fHiL+1911VVcccUVjB49mrFjxzJp0iQAxowZw7hx4xg5ciQDBw482OUEMGfOHGbOnElubi4LFizgkUce4aKLLqKqqgqAW2+9laFDh0bupIOI81sdwS9wEsYrzXRJvQLcrqofuOvvADeo6mEXJovIHJxWCBNyPRMe+dOt7M7puL67zqqsrOzg9d7dndVFg65YF+np6QwePDjs/UJ9pndXs27dOoqLiw/ZNn369CWqOrGtx4z2oHdTlxI0mcFUdS4wF2BiH5+OyqiAgoIIhhYbCgsLKbB6AKwugnXFuli9enWbxiIi9YjWzi4hIYFx48a16zGjfR9GEdAvaD0P2NraTgFvPBS1fHekMcaY9hXthPES8ENxTAGKWxu/ACiti4etX0CgNvIRGmM6jUh3oXcVkaqnSF9W+xRQAPQQkSLgJsAPoKr3A6/iXFK7Duey2itCOW5pXTzUlMOu1c7NNsaYLi8hIYE9e/aQnZ3d7I1xxkkWe/bsISEhod2PHemrpC5q5XMFfhruccuJB8qh6DNLGMZ0E3l5eRQVFbFr166w9qusrIzIj2dnlpCQQF5eXrsfN9qD3m1SjZ9KfyYJRUtg4r9FOxxjTAfw+/3k5+eHvV9hYWG7D/52V9Eew2gTr8D6+OGwxQa+jTGmo8RkwojzCp/W5MOuNVBZ3PoOxhhjjlhMJox4L7xb2h9Q2NLq1FPGGGPaQcwmjKV1g5wV65YyxpgOEaMJQyghmX1JA+wGPmOM6SAxmTA8AnmZiazxDnMSht3MY4wxEReTCQNgTF4G71cMgPLdsG9jtMMxxpguL3YTRr90Fhw4ylmxbiljjIm4mE0Yo/MyWKP9qPUlw+ZPoh2OMcZ0eTGbMEb1TadOvGxNHmEJwxhjOkDMJoyUeB+Dc1L4gqGwYwVUlUU7JGOM6dJiNmEAjOmXwVulA0DrYMuSaIdjjDFdWmwnjLx0FpYPcFY2fxrVWIwxpquL6YQxrn8mJSRTkjrYxjGMMSbCYjphHJ2bRmqCj9W+o6HoU6iri3ZIxhjTZcV0wvB6hEkDslhQnu/MWrv762iHZIwxXVZMJwyAKQOzeaPEvYHPuqWMMSZiYj5hTB6YxQbtTVVchg18G2NMBMV8whiRm0ZKvJ/1CSOthWGMMREU8wnD5/Vw7IBMPqgcCHvWwoE90Q7JGGO6pJhPGACTB2bzdukAZ6Xos6jGYowxXVXXSBj5WSzTgdSJDzZ9HO1wjDGmS+oSCWNU33S8cUlsShoB6xdEOxxjjOmSIp4wRGSGiKwRkXUicmMTn6eLyMsiskxEVorIFeF+h9/rYcKALBbUjoZty6BsZ/sEb4wx5qCIJgwR8QL3ADOBEcBFIjKiUbGfAqtUdQxQAPxBROLC/a7J+Vk8VzLcWfnGWhnGGNPeIt3CmASsU9X1qloNzAPOaVRGgVQRESAF2AvUhvtFUwZmsUIHUB2XCd+8c6RxG2OMacQX4eP3BTYHrRcBkxuV+SvwErAVSAUuVNXDJoUSkTnAHICcnBwKCwsP+by2TvF7PSyRkUxY/TofLXgXpEsM0bSorKzssLrorqwuGlhdNLC6aD+RThjSxDZttP4dYClwMjAIeEtE3lfVkkN2Up0LzAUYNmyYFhQUHHbgEzZ9xrtbJzC15gMKhmVBn7FHfAKdXWFhIU3VRXdkddHA6qKB1UX7ifSf4EVAv6D1PJyWRLArgOfUsQ7YAAxvy5cVDMvhhZJhzsq6t9tyCGOMMc2IdML4DBgiIvnuQPZsnO6nYJuAUwBEpBcwDFjfli8rGNaTXWSwJ2UYfPPuEYRtjDGmsYgmDFWtBa4G3gBWA8+o6koRuVJErnSL/Q9wnIh8CbwD3KCqu9vyff2ykhiYk8xHnnHOvFKVJa3vZIwxJiSRHsNAVV8FXm207f6g91uB09vr+6YP68nTnwzhLG8tbFgIR5/ZXoc2xphurctdRlQwLIdPaoZQ60u2y2uNMaYddbmEMSk/C58/nrXJ42Dt26CNL8oyxhjTFl0uYcT7vBw3KJt/lR8DxZtg15poh2SMMV1Cl0sY4HRL/bN0pLPy9WvRDcYYY7qILpowerKDLHanDoev34h2OMYY0yV0yYTRLyuJQTnJvC8TnMtry/dGOyRjjIl5XTJhAJw8vCdP7D0atM7u+jbGmHbQZRPGjFG5LKkdQGVcNnz9erTDMcaYmBdywhCR80Uk1X3/GxF5TkTGRy60IzOuXwa905NYHHes08II1EQ7JGOMiWnhtDD+S1VLReR4nBlmHwXui0xYR87jEWaOymVe8QioLIZNi6IdkjHGxLRwEkbAfZ0F3KeqLwJhPxmvI80a3ZsFNSOpE591SxljzBEKJ2FsEZG/ARcAr4pIfJj7d7hx/TJJTctkdcIYu7zWGGOOUDg/+BfgzDo7Q1X3A1nALyIRVHvxeISZx/TmubJRsGct7F4X7ZCMMSZmhZMwcoF/qepaESkAzgc+jURQ7emMY3J5rWYCisCX/4x2OMYYE7PCSRjPAgERGQw8BOQDT0YkqnY0oX8mgbS+rE4cB0ufhLrDHhdujDEmBOEkjDr3gUjnAner6nU4rY5Orf5qqQfLjnMmI9z4frRDMsaYmBROwqgRkYuAHwKvuNv87R9S+5s1Opd/1UykxpfitDKMMcaELZyEcQUwFbhNVTeISD7weGTCal8Tj8qkd3YG78WdCKtetEe3GmNMG4ScMFR1FXA98KWIjAKKVPX2iEXWjkSE88bncc++yVBbASufj3ZIxhgTc8KZGqQAWAvcA9wLfC0iJ0YmrPb3/Ql5LGUwexIHWLeUMca0QThdUn8ATlfVk1T1RJzpQf4YmbDaX5+MRI4fnMO8mhNh8yK7J8MYY8IUTsLwq+rB552q6tfEyKB3vfMm5PFI2WRUPLD0iWiHY4wxMSWchLFYRB4SkQJ3eQBYEqnAIuE7I3tTmZDDquTJsGwe1AVa38kYYwwQXsK4ClgJXANcC6wCrmxtJxGZISJrRGSdiNzYTJkCEVkqIitF5L0wYgpLgt/L2WP68LfiKVC6Fb5ZEKmvMsaYLscXakFVrQLucpeQiIgXZ5D8NKAI+ExEXnKvuKovk4EziD5DVTeJSM9Qj98W50/sx/mfjKMqIYP4pY/DkFMj+XXGGNNltJowRORLQJv7XFVHt7D7JGCdqq53jzUPOAendVLvB8BzqrrJPd7OEOJuszF56QzslcnrlSdw9lf/Qsr3QlJWJL/SGGO6hFBaGGcewfH7ApuD1ouAyY3KDAX8IlIIpAJ/UtXHGh9IROYAcwBycnIoLCxsc1BTe9Rw/6rjOCf+ZdY+979syZvV5mNFW1lZ2RHVRVdiddHA6qKB1UX7aTVhqOq3R3B8aeqQTcQwATgFSAQ+FpFF7lVYwXHMBeYCDBs2TAsKCtoc1OTqAFNvVzb5BzPkwCcMKbijzceKtsLCQo6kLroSq4sGVhcNrC7aTzg37pWKSEmjZbOIPC8iA5vZrQjoF7SeB2xtoszrqnpAVXcDC4Ex4ZxEuBLjvMw+tj8Pl02Dbctg+4pIfp0xxnQJ4VwldRfOA5P64vzwXw88AMwDHm5mn8+AISKSLyJxwGzgpUZlXgROEBGfiCThdFmtDiOuNrl06lG8rNOoFb/dk2GMMSEIJ2HMUNW/qWqpqpa4XURnqOrTQGZTO7jToV+N86S+1cAzqrpSRK4UkSvdMquB14HlOA9kelBVI/4nf9+MRCaPHMw7OgFdNg+qyyP9lcYYE9PCeh6GiFwgIh53uSDos5auonpVVYeq6iBVvc3ddr+q3h9U5g5VHaGqo1T17rDPoo2umJbPQ1WnIRV7YZnNL2WMMS0JJ2FcDFwK7HSXS4FLRCQRpxURcyYelUl57iRWe4eiH/3V7vw2xpgWhDO9+XpVPUtVe7jLWaq6TlUrVPWDSAYZKSLCj08cxN0VZyD7NsDql6MdkjHGdFrhXCWV514RtVNEdojIsyKSF8ngOsKsY3JZm3ECWzy56Id/Am22d80YY7q1cLqk/o5zhVMfnCulXna3xTSf18OVJw/jnqozkK2fw7cfRjskY4zplMJJGDmq+ndVrXWXR4CcCMXVob43ri+LUk9jv6Q7rQxjjDGHCSdh7BaRS0TE6y6XAHsiFVhH8ns9/J/pI3iw+nRk7ZuwY1XrOxljTDcTTsL4N+ACYDuwDTjP3dYlnDchj7eSz6RSEtCPrJVhjDGNhXOV1CZVPVtVc1S1p6p+9wjnmepU4n1eLjppDE/UTEeXz4f9m1vfyRhjupFQpjf/Cy3fmHdNu0YURbMn9ecHC7/HZVVvIh/fg8y8PdohGWNMpxHK9OaLIx5FJ5Hg93LpjGm8+NxUzln8CL6TfmnPyjDGGFco05s/GsqBROQvqvqzIw8pus4Z05efFc7m+8UfULNoLv6Tm3yqrDHGdDvhDHq3Zlo7HitqPB7h4rNm8m5gLLUf32eTEhpjjKs9E0aXcdzgHnzS51ISa/ZT/skj0Q7HGGM6BUsYzfj+9y5gcd1Q6t67A6pKox2OMcZEXXsmjKYexxqzhvZOY8mw60mp3cuu1/432uEYY0zUtWfC6HJ3u80+91xelRNJXzqXwJ4N0Q7HGGOiqtWEISJ3u68vi8hLjZf6cu7cUl1KeqIf7+k3E1Bh8zO/jHY4xhgTVaHch/EP9/XOSAbSWZ0+ZTzPf3wh5+74B3tXFZI1oiDaIRljTFS02sJQ1SXu27Gq+l7wAoyNaHSdgIgw4aKb2KZZlL14PdTVRTskY4yJinDGMC5rYtvl7RRHp3ZUbg5fDr+O/lVrWfav+6IdjjHGREUoYxgXicjLQH6j8YsFdJHpzUMx/fyfsMY3jD5L7mDXnm5z2sYYc1AoYxgf4Uxn3gP4Q9D2UmB5JILqjPw+H0ln/z9ynjuHlx/7L878+T2IdKkriY0xpkWhzCX1LfAtMDXy4XRu/UYX8M2HMzht+zO88v5lnHXi5GiHZIwxHSaULqlSESlpYikVkZKOCLIzyZ99Bx4RPO/cwqY9Ns+UMab7COUqqVRVTWtiSVXVtNb2F5EZIrJGRNaJSLNTv4rIsSISEJHzwj2JjuTJ7E/lpJ8ySz7k3kcfobImEO2QjDGmQ4R8lZSI9G9qaWUfL3APMBMYAVwkIiOaKfd74I3wwo+OtFN/QXnKUVxTfAd3PP9xtMMxxpgOEc5ltf8KWt4B1gOvtbLPJGCdqq5X1WpgHnBOE+V+BjwL7AwjnuiJSybpB4/R01PK1C9/w7OL7XGuxpiuL5SrpABQ1WOC10VkPPDvrezWFwj+NS0CDhkpFpG+wPeAk4FjmzuQiMwB5gDk5ORQWFgYaugRkzvwMk795kFue+H3/GPbufRL7fjJf8vKyjpFXXQGVhcNrC4aWF20n5ATRmOq+rmINPsD72rqutPGzwe/G7hBVQMtXaaqqnOBuQDDhg3TgoKC0IONFD2JqseL+OU3T3HVitHc/rPL6JES36EhFBYW0inqohOwumhgddHA6qL9hJwwROT/Bq16gAnArlZ2KwL6Ba3nAVsblZkIzHOTRQ/gDBGpVdUXQo0takSI//59VN8zjd+U/YGfPTqIv88pIMHvjXZkxhjT7sLpQ0kFUtwlDngZOLuVfT4DhohIvojEAbOBl4ILqGq+qg5Q1QHAfOAnMZEs6iVlEXf+AxwlO5i17R6u/+cy6uoaN6KMMSb2hZMwXgXG4Yw3XAT8CichNEtVa4Grca5+Wg08o6orReRKEbmybSF3QgOOR467mkt871C24lXueuvraEdkjDHtLpwxjMeB64EVQMhTtqrqqzjJJnjb/c2UvTyMeDqXk/8LXfcOf9n7ECctGESPlDgun5Yf7aiMMabdhJMwdqnqyxGLJNb54pFzHyDlgek8lPUPvvdyKknxPi6Y2K/1fY0xJgaEkzBuEpEHce7BqKrfqKrPtXtUsar3KOSUmxj35q/5Q89B/OJZITnOx6zRudGOzBhjjlg4CeMKYDjgp6FLSgFLGMGm/hR2rOD7yx5jc05vrp0nxPk8nDaiV7QjM8aYIxJOwhjT+OY90wQROOvPsH8z1xbdzfacW7nqcfjjhWM5a0yfaEdnjDFtFs5VUouamgfKNMEXBxf+A8noz++qb2dmnwqunfcF85cURTsyY4xps3ASxvHAUnfm2eUi8qWIdJsHKIUtKQt+8Awe4E/6v5w2MIHr/7mMRz/aGO3IjDGmTcLpkpoRsSi6quxBcOHjeB47h3uP+jNXDf9PbnppJVv2V3DjjOF4PPbEPmNM7Ai5haGq3za1RDK4LmHANDjzj3g3FHJ/j6f54dSjmLtwPT998nN7loYxJqZ0/PSq3dH4S+G4n+FZ/BC39FzIb2YdzesrtzN77iJ2lFRGOzpjjAmJJYyOcuotMPxM5I1f8aOkhdx38QS+3lHKrD9/wCfr90Q7OmOMaZUljI7i8cJ5D8Pg0+DlnzOj5m1e+Ok0UhN8/ODBT3jogw2o2qSFxpjOyxJGR/LFw4WPw8ACePFqhm7/Fy9ePY2Th/fkf15ZxZWPL2HfgepoR2mMMU2yhNHR/Alw0VOQfwI8fyVpq+bxt0sm8Oszjubdr3Yy408L+WDt7mhHaYwxh7GEEQ3+RLjoaRh0Mrx0NZ5P/8aPTxzI8z+ZRmqCn0se+oRbXl5JeXVttCM1xpiDLGFES1yS09I4+ix4/QZYeAej+qTx8tXH88OpR/H3DzfynbsX8uE6a20YYzoHSxjR5IuH8x6B0bPh3VvhtRtI9Cr/fc4onp4zBZ/Hw8UPfsIv5y9jr41tGGOizBJGtHl98N37YOrV8Onf4MkLoLKYyQOzee3aE7jypEE8+/kWpt9ZyGMfb6Q2EPKzq4wxpl1ZwugMPB74zm3OLLcb3oMHT4O960nwe7lx5nBeu/YERvZJ47cvruTMv3xg3VTGmKiwhNGZTLgMLn0BDuyE+0+Ezx8DVYb2SuWJH03m/kvGU1pZy8UPfsIPH/6UlVuLox2xMaYbsYTR2eSfAHPegz5j4aWfwRPnQ8lWRIQZo3J55z9O4jezjmbZ5v2c+ZcPuH9ZJet2lkY7amNMN2AJozPKPAp++BLMvAO+/RDumQIf/RVqq0nwe/nRCQNZ+Mvp/PuJg/h8Z4DT/riQnz7xOau3lUQ7cmNMF2YJo7PyeGDyHLjyA+h3LLz5a7h3Mqx+GVRJT/Rz48zh3HlSEledNIjCNTuZ+af3ufzvn/Lhut02zYgxpt1ZwujssgfBJc/Cxc+CNw6evgSeugjKdgGQFif8csZwPrzxZP7vaUNZsaWYix/8hFl//oB/Lt5sU6gbY9qNJYxYMeRUuPJDOP02+OZduG8qfP3GwY8zkuK45pQhfHDDydx+7jFUB+r4xfzlTP7dO9z2r1Vs2H0gisEbY7qCiCcMEZnhPtZ1nYjc2MTnF7uPfF0uIh+JyJhIxxSzvD447mqYUwgpveDJCxi++m7Yve5gkQS/l9mT+vPWdSfy1I+ncPzgHvz9w41Mv7OQC+7/mH8u3syBKptyxBgTvnAe0Ro2EfEC9wCnAUXAZyLykqquCiq2AThJVfeJyExgLjA5knHFvF4j4MfvwoLf0fOje+CvE50pRo7/OfSdAICIMHVQNlMHZbOzpJL5nxfxz8VF/GL+cm56aSWnj+jF2WP7cMKQHPxea2gaY1oX0YQBTALWqep6ABGZB5wDHEwYqvpRUPlFQF6EY+oafPFw2i18rOOY5l0Onz4Iq1+C/BNh2rUw6BQQ55nhPdMS+EnBYK46aRBLvt3Hs58X8eqX23lh6VYykvx8Z0RvZozqzXGDs4n3eaN8YsaYzkoieTWNiJwHzFDVH7nrlwKTVfXqZspfDwyvL9/osznAHICcnJwJzzzzTMTijiVlZWWkpKTgrS2nz9Y3yCt6mfjqPZQlD2DjgAvZ3WPqwcQRrLZOWbE7wCfbalm6K0BFLST6YHQPL2N6+jimh5fUuMP368zq68JYXQSzumgwffr0Jao6sa37RzphnA98p1HCmKSqP2ui7HTgXuB4VW3xmaXDhg3TNWvWRCLkmFNYWEhBQUHDhtpq+PKf8OHdsPtr6DMeTr0ZBp7U7DGqagN8tG4Pr63Yxrtf7WR3WTUegbH9MjhhSA7HD+nB2H4Znb7r6rC66MasLhpYXTQQkSNKGJHukioC+gWt5wFbGxcSkdHAg8DM1pKFaYUvDsZdDGNmw7KnYMH/wmNnQ78pzjjH0BnQY/Ahu8T7vEwf3pPpw3tSV6d8uaWYd7/aSeHXu/jLu2v50ztrSY7zcmx+FpPzs5kyMItRfdM7fQIxxrSvSCeMz4AhIpIPbAFmAz8ILiAi/YHngEtV9esIx9N9eLww7hIYdR4sfgi+eNy5+e/NX0P2EBj5XRj1feh59KG7eYQx/TIY0y+D604bSnF5DR+v38MH63axaP1eCtd8BUBSnJfx/TOZlJ/FsQOyGJ2XTnJ8pP93MsZEU0T/hatqrYhcDbwBeIGHVXWliFzpfn4/8FsgG7hXnL722iNpMplG/Akw9afOsu9b596Nr16B9/8AC++AniNg9IUw5iJI7XXY7ulJfmaMcgbFAXaVVvHphr18smEPn27Yyx/f/hpV8AgM7ZXKmLwMjslL55i+6QzrnUqC3wbRjekqIv4noaq+CrzaaNv9Qe9/BBw2yG0iIPMoZ7qRyXOgbCesetEZ73j7Jnjnv53uqrEXwcACiE9t8hA5qfHMGp3LrNG5ABSX17Bk016Wbi5m2eb9vLlqO08v3gyA1yMM6ZnCiNw0js5NY0SfNIb1TqVHSnxHnbExph1ZH0J3ldITJv3YWXZ9DV/8wxnzWPMv8Pih/xQYcjoMn+VMT9KM9CQ/Jw/vxcnDndaJqlK0r4KVW4tZsaWEFVuL+fCb3Tz3xZaD+2QnxzG0VyrDeqcyqGcKQ3qmMLhnCtnJcUgTV3QZYzoHSxgGcobC6f8Dp/wWNn0Ma9+CdW/DW//lLD1HOgPmI7972JhHYyJCv6wk+mUlMWNU7sHte8qqWL2tlDU7Svl6eylf7Sh17jqvbpjrKi3Bx8CcFAbmJJOfncyAHsnk90jmqOwkUhP8kTp7Y0yILGGYBl6/c+Nf/olOAtm/CVa/4syQ+97v4b3bnTGPUefC0edAjyFN3uPRlOyUeI4fEs/xQ3oc3KaqbCuuZO3OMtbtLGPD7jLW7zrAR+v28NznWw7ZPzPJT//sZPpnJdEvM5G8zCT6ZSXSNyORPhmJNlZiTAewhGGal9Efpv7EWUp3OGMeK56Fd291lqQe0G8y9JsEuaOh1yinqytEIkIf9wf/pKE5h3xWXl3Lt3vK2bj7AN/uLefbPeVs3lvO0s37eO3LbdTWHXr/UI+UeFK9NczbvITcjARy0xPolZZAbnoivdMS6JkWb0nFmCNkCcOEJrVXw4D5/s1Ol9XmT2HzImfco15SD8ib2NBS6TnSebZHmJLifBztDpY3FqhTtpdUUrS3nC37K9iyr4It+ytYsX4r63aV8f7aXYd0ddXLSPLTK9VJHjmp8fRMTSAn1XmfkxJPTmocPVLiSU/021iKMU2whGHCl9EPJl7hLAAH9sCOFbBzFWxf4YyDfP2681l8GmQNdJbswZB3rDOgnnB4IgiV1yP0zXC6o4IVFu6loOAkVJWSylp2lFSyvbiS7SWV7CypZEdJFdtLKtlVWsX6XQfYVVpFdaDusOP7PEJWchzZKfFkJ8e57+PISoojy33NSHK2Zyb5yUiKI85nNzGars8ShjlyydnO1CPB048Ub4GN78OWJbDnG9j6Oax6AbQOxAO5Y5wloz9kHAWZ+c6AelzSEYcjIqQn+klP9DO0V9OXB4MzhrK/vIbdZVXsKqtiV2kVu8uq2VNWxZ6yavYcqGLPgWo27ytnb1k1pS1MC58c5yUjKY7MZD8ZiXGkJ/nJcGPISPK78cSRkRS87ifR77XWjIkZljBMZKT3daYnGTO7YVt1ORR9Chs/dJ5VvvoVKN/d8Ll4nLvQe49yEkh6X0jv57ROMgc4d6+3IxEhMzmOzOQ4hrSQWA6GX1vHvvJq9pRVs7+8mr3l1ewrr2HfgWr2l9ewv7yafeXVFFfUsLW4gv3lNRRX1BCoa36+Nr/XSW5pCX7SEp0lNcHnrjuvqQk+Z4l33qc0em9TtJiOYgnDdJy4JOemwIEFDduqDzhjInvWOt1Z27+Eos9g5QugQeMQvgTIGQY9hkJqLqT2dpbMfOdqrWZuNGzX8H0eeqU5g+mhUlUOVAfYX+4klZKKGvZX1BxMJiWV7mtFDSWVtZRU1FC0t5ySylpKK2uoqj28y6yxeJ+H1AQ/KfFeUhJ8JMf5qCyr5LltX5CS4CMl3tmWHO8lOd5HcryPlHgvSXHOZ0lxzvakOGeb12MtHtM0SxgmuuKSoedwZzn6rIbtdQEo3Q7FRbBnnTM+snM1bPoEyrZDoPrQ46T0ZpwnHYr6Q0I6JOe4xx3pdHXFR2d6axEhJd75Yc7LDH//ypoAZVW1lLoJxHlteH+gqpayqlpK3PcHqmopraplf5Wyv2i/s15ZG1LiqRfv8wQlEC+JcT6S/PXv3Ve/u91dD044weWS/D4S3fVEv9eSUYyzhGE6J4/X7ZLqC/0bPYBRFSr2OQll7zewey3sXktg0yqni2vPOmfqk5qg55in5jrdWhlHOVd8+ZOdFk9cCqT1dQby0/M6pKUSjgS/lwS/N+zpVBpP6V0bqKO8JuAmlcDB5HKgOkB5dcO28vr1aud9RXWAA9UBKqpr2V5SQ0V14GCZipoANYHwHo8Q5/M4ycbvJJEEv5dEv8d99ZIQ5yXB5yXB3Zbg97jrzvt4tz4SfJ6DdRPvaygb72tYj/N5LEG1M0sYJvaIQFKWs/QacXDz8uAfybo6KN4EO1bCjlWwbwPs2wgbP4ADuyBQ1fSxPT6n1ROX4iSPxExnSUgHfyL4Ep0JHeNTnW0JGU4caX0hrY+zbyfk83pI83pIa+c75msCdUGJpZbyKieZlNcEKK8KUFHjJJvyavd9TYBKN+lU1tZRUR2gssZZiitqqKgJUFVTR2V92ZoALQwBtcrvFbwoye+/RbzPSTjxPo+7eIn3O+/j3PU4r4d4v4c4r7OtfntDmUNf47xOYvJ7pWG7uy14e5zX0yUubrCEYbomj8dpUWQOcObDaixQCzXlUFXiXNFVvNnp/qosdsZVag5AZYnTktm/GSq/hJoKqK10XvXw+zwAiE+HpPokk+EkEF88eOOdRBOXDHGpzmt8qnN5cXyaUz4p21nikkO+gz7a/F4P6Yke0hMjM3WLqlITUCprneRRn0wqa+qoqnVeK2sCVNYGqK6tO7heHahzytYGWL/hW3Jye7vrdVTXBqiqdcodqKpl74GGfaprnaXKfW18g+iR8HsFv7c+kThJpH6b3+vB7/MQF7we9N7nFbd8w3tfE2V9XucYPk9DufFHZYY17tYSSxime/L6wJvm/GCn5wGTW93lIFUnaVQWQ+V+p8VSsg1KtkDpNifJVOyDiv1QtgNqq5wxl5oKJxnVVrQSW5yTbBIznFZMfGrD4ncTkD/RKefxNSwJ6QdbREkHipwuu4R054KBGElAjYkIcT7nr/S2to4KC7dTUHBMm/YN1GlDEgk4Cas+mQQnmOpAwH1tKF9d63TZ1ZerCTTsVxNQagIN2+rX68sdqHaOVxtwklb99to6pcY9Rm2dtngFXr25l07g9JG923T+jVnCMCZcIu74RxKk5QItT8h4mLoAVJdBVanTiqlyWzLle+DAbqjY6yajYifpVLqtoKoSp1VUW+W0dFowCZzHl4GTTLzxzlxhvni3Gy3b6UpLSG9oAfniQLzO+JF4ne1xyeBPchKUP9FNVo1aR/WXO6sC6r7W15XHWTzeoO3qbPPGHXqpdF0d1NU43+3tHD9NXo8cHLSHzjcBZl2dUlNXR219wnGTUW1QUuqbmdj6gULUOf6rGNOdeLzu+Ec6pLfxGKpOq6UuAHW1zlJZ7CSb8n2s+vwjRgzs62yrKnXKBqqdRFOxH8r3OhcHVJU54zm17qIB55i0X1dMi8TrJLL6cwje7ktwklTwOJJ4OJiUtM59714B5o1rWOJTnGQWn0a/TVvgg6VuvdVBoMatuxqnrC/eHZtKdBJhXLLzPjh5qpvM6mqdqvG6rbr6zzTgvNYnZ5/b+qsLNMTp8Tvn6vE1/DfUACANydwb5x6vztlXPG4L0usm36CWonjwiJd4j5d4j4IEwBMAv7qxuft52u9n3hKGMbFIxPmBCZaUBeQDsHOLjxHHFrT9+KpOcqkud1o1NRVOV1pNZcPYT33rqC7g/pBJQ2wIDT/sgcPLBP9wB6qdH8z6Ljata2hFVR9wuv0q9jldf6hzDJGG1ot43AS63zlm/X5ui2wQwPrG9ef+kAaq6bDkGC2zn2x6HK8NLGEYYw4n0tANRXa0o2m7QA0LC9/lxBNOcNZFDu0KU3VaDTUVTiKsPuC0yGrKG/7K10BDgvG63VL1LaL6VkD9UlfrtuSqnPf1LRRwWiiBmoaWlHgbWgz1+wRqnG31LYr6+OrjqFffwgpOxvUtIsTdXut08/UY1m7VaQnDGNN1ef3UeeObn6NM3O4gr/+IJsTsLmwSGmOMMSGxhGGMMSYkljCMMcaExBKGMcaYkFjCMMYYE5KIJwwRmSEia0RknYjc2MTnIiJ/dj9fLiLjIx2TMcaY8EU0YYiIF7gHmAmMAC4SkRGNis0EhrjLHOC+SMZkjDGmbSLdwpgErFPV9apaDcwDzmlU5hzgMXUsAjJEJDfCcRljjAlTpG/c6wtsDlov4vBpQZsq0xfYFlxIRObgtEDIycmhsLCwvWONSWVlZVYXLquLBlYXDawu2k+kE0ZTcyo3nrgllDKo6lxgLsCwYcM0+Gli3VnjJ6t1Z1YXDawuGlhdtJ9Id0kVAf2C1vOArW0oY4wxJsoinTA+A4aISL6IxAGzgZcalXkJ+KF7tdQUoFhVtzU+kDHGmOiKaJeUqtaKyNXAG4AXeFhVV4rIle7n9wOvAmcA64By4IpIxmSMMaZtIj5braq+ipMUgrfdH/RegZ9GOg5jjDFHxu70NsYYExJLGMYYY0JiCcMYY0xILGEYY4wJiSUMY4wxIbGEYYwxJiSWMIwxxoTEEoYxxpiQWMIwxhgTEksYxhhjQmIJwxhjTEgsYRhjjAmJJQxjjDEhsYRhjDEmJJYwjDHGhEScx1HEFhEpBdZEO45OogewO9pBdBJWFw2sLhpYXTQYpqqpbd054g9QipA1qjox2kF0BiKy2OrCYXXRwOqigdVFAxFZfCT7W5eUMcaYkFjCMMYYE5JYTRhzox1AJ2J10cDqooHVRQOriwZHVBcxOehtjDGm48VqC8MYY0wHs4RhjDEmJDGXMERkhoisEZF1InJjtOPpKCLST0QWiMhqEVkpIte627NE5C0RWeu+ZkY71o4iIl4R+UJEXnHXu2VdiEiGiMwXka/c/z+mduO6uM7997FCRJ4SkYTuUhci8rCI7BSRFUHbmj13EfmV+zu6RkS+E8p3xFTCEBEvcA8wExgBXCQiI6IbVYepBf5DVY8GpgA/dc/9RuAdVR0CvOOudxfXAquD1rtrXfwJeF1VhwNjcOqk29WFiPQFrgEmquoowAvMpvvUxSPAjEbbmjx397djNjDS3ede9/e1RTGVMIBJwDpVXa+q1cA84Jwox9QhVHWbqn7uvi/F+VHoi3P+j7rFHgW+G5UAO5iI5AGzgAeDNne7uhCRNOBE4CEAVa1W1f10w7pw+YBEEfEBScBWukldqOpCYG+jzc2d+znAPFWtUtUNwDqc39cWxVrC6AtsDlovcrd1KyIyABgHfAL0UtVt4CQVoGcUQ+tIdwO/BOqCtnXHuhgI7AL+7nbPPSgiyXTDulDVLcCdwCZgG1Csqm/SDesiSHPn3qbf0lhLGNLEtm51XbCIpADPAj9X1ZJoxxMNInImsFNVl0Q7lk7AB4wH7lPVccABum6XS4vc/vlzgHygD5AsIpdEN6pOq02/pbGWMIqAfkHreThNzm5BRPw4yeIJVX3O3bxDRHLdz3OBndGKrwNNA84WkY043ZIni8jjdM+6KAKKVPUTd30+TgLpjnVxKrBBVXepag3wHHAc3bMu6jV37m36LY21hPEZMERE8kUkDmfQ5qUox9QhRERw+qlXq+pdQR+9BFzmvr8MeLGjY+toqvorVc1T1QE4/w+8q6qX0D3rYjuwWUSGuZtOAVbRDesCpytqiogkuf9eTsEZ6+uOdVGvuXN/CZgtIvEikg8MAT5t7WAxd6e3iJyB03/tBR5W1duiG1HHEJHjgfeBL2not/9PnHGMZ4D+OP9gzlfVxgNfXZaIFADXq+qZIpJNN6wLERmLM/gfB6wHrsD5Y7A71sUtwIU4VxV+AfwISKEb1IWIPAUU4EznvgO4CXiBZs5dRH4N/BtOXf1cVV9r9TtiLWEYY4yJjljrkjLGGBMlljCMMcaExBKGMcaYkFjCMMYYExJLGMYYY0JiCcN0SyJS5r4OEJEftPOx/7PR+kfteXxjosUShunuBgBhJYwQZvU8JGGo6nFhxmRMp2QJw3R3twMniMhS91kKXhG5Q0Q+E5HlIvLv4Nwg6D6P5EmcmycRkRdEZIn7/IU57rbbcWZLXSoiT7jb6lsz4h57hYh8KSIXBh27MOiZFk+4dyojIreLyCo3ljs7vHaMCeKLdgDGRNmNuHeKA7g//MWqeqyIxAMfisibbtlJwCh3OmiAf1PVvSKSCHwmIs+q6o0icrWqjm3iu84FxuI8s6KHu89C97NxOM8m2Ap8CEwTkVXA94DhqqoiktG+p25MeKyFYcyhTgd+KCJLcaZdycaZZwfg06BkAXCNiCwDFuFM5DaElh0PPKWqAVXdAbwHHBt07CJVrQOW4nSVlQCVwIMici5QfoTnZswRsYRhzKEE+JmqjnWXfPeZCuBMHe4UcuawOhWYqqpjcOYtSgjh2M2pCnofAHyqWovTqnkW58E3r4dxHsa0O0sYprsrBVKD1t8ArnKnkkdEhroPJGosHdinquUiMhznsbn1aur3b2QhcKE7TpKD86S8ZmcIdZ99kq6qrwI/x+nOMiZqbAzDdHfLgVq3a+kRnOdjDwA+dweed9H0Iz1fB64UkeXAGpxuqXpzgeUi8rmqXhy0/XlgKrAM52E1v1TV7W7CaUoq8KKIJOC0Tq5r0xka005stlpjjDEhsS4pY4wxIbGEYYwxJiSWMIwxxoTEEoYxxpiQWMIwxhgTEksYxhhjQmIJwxhjTEj+Pxwp/REFvNwiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#因此可以尝试将lambda_l2设置为0.9\n",
    "lgb.plot_metric(results)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaMAAAEWCAYAAADLkvgyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjUElEQVR4nO3dfXhV5Znv8e8PUBGwKAURoRgFUREUBN9aDo1VrCjV2rFaq1aqI+P0OGqrqNN2WlodUaszzkjPOFitGe2gVouitVQPGj1HZzyCioCVKdZ0AHmpLyBQRQL3+WOt6E5Iwg5h82Qnv8917StrP896ue9Acmc969lrKSIwMzNLqVPqAMzMzFyMzMwsORcjMzNLzsXIzMySczEyM7PkXIzMzCw5FyOzMiLpu5J+ljoOsx1N/pyRdRSSaoC+wOaC5iER8VYr9/mXEfG/Wxdd+ZE0BRgcEeemjsXKn8+MrKP5UkT0KHhtdyHaESR1SXn87VWucVvb5WJkHZ6knpLulLRC0nJJ10nqnPcNkvSUpHckvS3pF5L2zPvuAQYCj0paL+kqSZWSljXYf42kE/LlKZIelHSvpPeBic0dv5FYp0i6N1+ukBSSvilpqaT3JF0s6UhJr0paI2lawbYTJT0n6TZJayW9Lun4gv59Jc2S9K6kJZIuanDcwrgvBr4LnJXnPj9f75uSfidpnaQ/SPqrgn1USlom6QpJq/N8v1nQv7ukWyT9MY/v/0raPe87RtLzeU7zJVVuxz+1tWEuRmZQBdQCg4GRwInAX+Z9AqYC+wKHAJ8BpgBExHnAf/PJ2dZNRR7vNOBBYE/gF9s4fjGOBg4EzgJuBb4HnAAcCpwp6fMN1v0D0Bv4IfArSb3yvhnAsjzXM4DrC4tVg7jvBK4H7s9zPzxfZzUwAfgU8E3gHyUdUbCPfYCeQH/gQuCnkvbK+24GRgGfBXoBVwFbJPUHfg1cl7dfCTwkqU8LvkfWxrkYWUfzcP7X9RpJD0vqC4wHLo+IDRGxGvhH4GsAEbEkIp6MiI0R8SfgH4DPN737ovxHRDwcEVvIfmk3efwiXRsRH0bEE8AGYEZErI6I5cD/IStwdVYDt0bEpoi4H1gMnCLpM8AY4Op8X68APwPOayzuiPigsUAi4tcR8UZkngGeAP5HwSqbgB/nx38cWA8cJKkTcAFwWUQsj4jNEfF8RGwEzgUej4jH82M/CcwFTm7B98jaOI/7Wkfz5cLJBpKOAnYBVkiqa+4ELM379wb+mewX6h5533utjGFpwfJ+zR2/SKsKlj9o5H2PgvfLo/6spT+SnQntC7wbEesa9I1uIu5GSRpPdsY1hCyPbsCCglXeiYjagvd/zuPrDXQF3mhkt/sBX5X0pYK2XYCntxWPlQ8XI+volgIbgd4NfknWmQoEcFhEvCPpy8C0gv6G01E3kP0CBiC/9tNwOKlwm20df0frL0kFBWkgMAt4C+glaY+CgjQQWF6wbcNc672XtBvwEPAN4JGI2CTpYbKhzm15G/gQGATMb9C3FLgnIi7aaitrNzxMZx1aRKwgG0q6RdKnJHXKJy3UDcXtQTaUtCa/djG5wS5WAQcUvP8voKukUyTtAnwf2K0Vx9/R9gYulbSLpK+SXQd7PCKWAs8DUyV1lXQY2TWdXzSzr1VART7EBrArWa5/Amrzs6QTiwkqH7K8C/iHfCJFZ0nH5gXuXuBLkr6Yt3fNJ0MMaHn61la5GJllf8nvCrxGNgT3INAv7/sRcASwluwi+q8abDsV+H5+DerKiFgLfIvsestysjOlZTSvuePvaC+QTXZ4G/h74IyIeCfvOxuoIDtLmgn8ML8+05Rf5l/fkfRSfkZ1KfAAWR5fJzvrKtaVZEN6LwLvAjcCnfJCeRrZ7L0/kZ0pTca/v9oVf+jVrIOQNJHsA7pjUsdi1pD/sjAzs+RcjMzMLDkP05mZWXI+MzIzs+T8OaPttOeee8bgwYNTh9FqGzZsoHv37qnDaJX2kAM4j7bGeZTGvHnz3o6IrW7l5GK0nfr27cvcuXNTh9Fq1dXVVFZWpg6jVdpDDuA82hrnURqS/thYu4fpzMwsORcjMzNLzsXIzMySczEyM7PkXIzMzCw5FyMzM0vOxcjMzJJzMTIzs+RcjMzMLDkXIzMzS87FyMzMknMxMjOz5FyMzMwsORcjMzNLzsXIzMySczEyM7PkXIzMzCw5FyMzM0vOxcjMzJJzMTIzs+RcjMzMLDkXIzMzS87FyMzMknMxMjOz5FyMzMwsORcjMzNLzsXIzMySczEyM7PkXIzMzCw5FyMzM0vOxcjMzJJzMTIzs+RcjMzMLDkXIzMzS87FyMzMknMxMjOz5FyMzMwsOUVE6hjK0sADBkenM/8pdRitdsXwWm5Z0CV1GK3SHnIA59HWtOc8am44JVE0IGleRIxu2O4zIzOzDu6CCy5g7733ZtiwYR+3TZ48mYMPPpjDDjuM008/nTVr1gDw5JNPMmrUKIYPH86oUaN46qmndkgMLkZmZh3cxIkTmT17dr22cePGsXDhQl599VWGDBnC1KlTAejduzePPvooCxYsoKqqivPOO2+HxFDSYiRpH0n3SXpD0muSHpc0pIl1KyQtLGU8TZE0W9J8SYsk3S6pc4o4zMxSGDt2LL169arXduKJJ9KlSza8d8wxx7Bs2TIARo4cyb777gvAoYceyocffsjGjRtbHUPJipEkATOB6ogYFBFDge8CfUt1zFY4MyIOB4YBfYCvJo7HzKzNuOuuuxg/fvxW7Q899BAjR45kt912a/UxSnl17jhgU0TcXtcQEa8o8xNgPBDAdRFxf+GGkiYCoyPikvz9Y8DNEVEtaT3wU+AE4D2yAncTMBC4PCJm5dufCnQDBgEzI+KqpgKNiPfzxS7ArnlcW5E0CZgE0Lt3H34wvLYF3462qe/u2QXOctYecgDn0da05zyqq6u3Wm/lypVs2LBhq757772XNWvW0L9//3p9b775Jt///ve56aabGt1fS5WyGA0D5jXS/hVgBHA40Bt4UdKzLdhvd7KzraslzQSuA8YBQ4EqYFa+3ghgJLARWCzptohY2tROJf0WOAr4DfBgY+tExHRgOmSz6drrTJty0x5yAOfR1rTnPGrOqdxqvZqaGrp3705l5Sd9VVVVLFq0iDlz5tCtW7eP25ctW8akSZN44IEH+NznPrdD4kwxgWEMMCMiNkfEKuAZ4MgWbP8RUHelbQHwTERsypcrCtabExFrI+JD4DVgv+Z2GhFfBPoBuwFfaEE8ZmbtzuzZs7nxxhuZNWtWvUK0Zs0aTjnlFKZOnbrDChGUthgtAkY10q4itq2lfmxdC5Y3xScfjtpCduZDRGyh/ple4RW1zRRxFpgXrlnAaUXEaGbWLpx99tkce+yxLF68mAEDBnDnnXdyySWXsG7dOsaNG8eIESO4+OKLAZg2bRpLlizh2muvZcSIEYwYMYLVq1e3PoiIKMmLrOi8AFxU0HYk8EPgt0BnsskCfwT2ITurWZivNwZ4nqwgfQZ4H6jM+9YX7G8KcGXB+/X514nAtIL2x+q2byTOHkC/fLkLcD9wybbyGzJkSLQHTz/9dOoQWq095BDhPNoa51EawNxo5HdqyQZEIyIknQ7cKuka4EOgBrg8LwDzySYKXBURKyVVFGz+HPAm2dDbQuClUsVJdg1qlqTdyArkU8DtzW9iZmY7UkmvzkXEW8CZjXRNzl+F69aQTXogr57nNLHPHgXLUxrri4i7gbsL2ic0E+MqWnbNyszMdjDfgcHMzJIr/3mLLSDpBbLZcoXOi4gFKeIxM7NMhypGEXF06hjMzGxrHqYzM7PkXIzMzCw5FyMzM0vOxcjMzJJzMTIzs+RcjMzMLDkXIzMzS87FyMzMknMxMjOz5FyMzMwsORcjMzNLzsXIzMySczEyM7PkXIzMzCw5FyMzM0vOxcjMzJJzMTIzs+RcjMzMLDkXIzMzS87FyMzMknMxMjOz5FyMzMwsORcjMzNLzsXIzMySczEyM7PkXIzMzCw5FyMzM0vOxcjMzJJzMTIzs+QUEaljKEsDDxgcnc78p9RhtNoVw2u5ZUGX1GG0SnvIAZxHW1OYR80Np3zcfsEFF/DYY4+x9957s3DhQgDeffddzjrrLGpqaqioqOCBBx5gr7324p133uGMM87gxRdfZOLEiUybNm2n51FdXU1lZeVOP25TJM2LiNEN231mZGbWAhMnTmT27Nn12m644QaOP/54fv/733P88cdzww03ANC1a1euvfZabr755hShlpWSFiNJ+0i6T9Ibkl6T9LikIU2sWyFpYSnjaeK43ST9WtLrkhZJumFnx2Bm5WPs2LH06tWrXtsjjzzC+eefD8D555/Pww8/DED37t0ZM2YMXbt23dlhlp2SFSNJAmYC1RExKCKGAt8F+pbqmK1wc0QcDIwEPidpfOqAzKx8rFq1in79+gHQr18/Vq9enTii8lPKgd3jgE0RcXtdQ0S8osxPgPFAANdFxP2FG0qaCIyOiEvy94+RFYxqSeuBnwInAO+RFbibgIHA5RExK9/+VKAbMAiYGRFXNRZkRPwZeDpf/kjSS8CAxtaVNAmYBNC7dx9+MLy25d+VNqbv7tnYeDlrDzmA82hrCvOorq6u17dy5Uo2bNjwcXttbW29dRq+f/3111m+fPlW+9kZ1q9fn+S4LVXKYjQMmNdI+1eAEcDhQG/gRUnPtmC/3cnOtq6WNBO4DhgHDAWqgFn5eiPIznQ2Aosl3RYRS5vbsaQ9gS8Bjc5MiIjpwHTIJjC0t4u05ao95ADOo62pN4HhnMp6fTU1NXTv3v3jiQH9+/fnoIMOol+/fqxYsYJ999233qSBmpoa1q9fn2QiQVubwNCUFBMYxgAzImJzRKwCngGObMH2HwF1Vw8XAM9ExKZ8uaJgvTkRsTYiPgReA/ZrbqeSugAzgH+OiD+0IB4z6+BOPfVUqqqqAKiqquK0005LHFH5KerPF0mDgGURsVFSJXAY8G8RsaaZzRYBZzS2uyIOWUv9Qll49W9TfDIffQvZmQ8RsSUvKHU2FixvZtu5Tgd+HxG3FhGfmXVQZ599NtXV1bz99tsMGDCAH/3oR1xzzTWceeaZ3HnnnQwcOJBf/vKXH69fUVHB+++/z0cffcTDDz/ME088wdChQxNm0DYVey79EDBa0mDgTrKhsH8HTm5mm6eA6yVdFBF3AEg6kuw6z1mSqoBewFhgMvULTg3wLUmdgP7AUUVntB0kXQf0BP6ylMcxs/I3Y8aMRtvnzJnTaHtNTU0Jo2k/ii1GWyKiVtLpwK0RcZukl5vbICKibn1J1wAfkhWZy4EewHyyCQxXRcRKSRUFmz8HvEk29LYQeKn4lFpG0gDge8DrwEvZJECmRcTPmttu9106s7jgg3Dlqrq6eqvx8HLTHnIA59HWtJc8ykWxxWiTpLOB88ku8APssq2NIuIt4MxGuibnr8J1a8gmPZAPw53TxD57FCxPaawvIu4G7i5on9BMjMsobujQzMxKpNgJDN8EjgX+PiLelLQ/cG/pwjIzs46kqDOjiHhN0tVkn+UhIt4Eyu5OBZJeAHZr0HxeRCxIEY+ZmWWKnU33JeBmYFdgf0kjgB9HxKkljG2Hi4ijU8dgZmZbK3aYbgrZjLY1kN1JAdi/JBGZmVmHU2wxqo2ItQ3a/OwJMzPbIYqdTbdQ0teBzpIOBC4Fni9dWGZm1pEUe2b0N8ChZHc1+HdgLdnnhczMzFptm2dGkjoDsyLiBLIPh5qZme1Q2zwziojNwJ8l9dwJ8ZiZWQdU7DWjD4EFkp4ENtQ1RsSlJYnKzMw6lGKL0a/zl5mZ2Q5X7B0YqkodiJmZdVzF3oHhTRr5XFFEHLDDIzIzsw6n2GG60QXLXYGvkj2LyMzMrNWK+pxRRLxT8FqePw31C6UNzczMOopih+mOKHjbiexMaY+SRGRmZh1OscN0txQs15I9hbWxh+aZmZm1WLHF6MKI+ENhQ/6APTMzs1Yr9t50DxbZZmZm1mLNnhlJOpjsBqk9JX2loOtTZLPqzMzMWm1bw3QHAROAPYEvFbSvAy4qUUxmZtbBNFuMIuIR4BFJx0bEf+ykmMzMrIMpdgLDy5L+J9mQ3cfDcxFxQUmiMjOzDqXYCQz3APsAXwSeAQaQDdWZmZm1WrHFaHBE/B2wIb9p6inA8NKFZWZmHUmxxWhT/nWNpGFAT6CiJBGZmVmHU+w1o+mS9gL+DpgF9AB+ULKozMysQyn2eUY/yxefAfzYCDMz26GKGqaT1FfSnZJ+k78fKunC0oZmZmYdRbHXjO4Gfgvsm7//L+DyEsRjZmYdULHFqHdEPABsAYiIWmBzyaIyM7MOpdhitEHSp8kfPS7pGGBtyaIyM7MOpdjZdN8hm0U3SNJzQB/gjJJFVQY+2LSZimt+nTqMVrtieC0TyzyPVDnU3HDKTj+mWXu1rbt2D4yI/46IlyR9nuzGqQIWR8Sm5rY164gqKirYY4896Ny5M126dGHu3LlMnjyZRx99lF133ZVBgwbx85//nD333DN1qGZtyraG6R4uWL4/IhZFxEIXIrOmPf3007zyyivMnTsXgHHjxrFw4UJeffVVhgwZwtSpUxNHaNb2bKsYqWC5xZ8vkrSPpPskvSHpNUmPSxrSxLoVkha29Bg7gqS/l7RU0voUx7f27cQTT6RLl2wQ4phjjmHZsmWJIzJre7ZVjKKJ5W2SJGAmUB0RgyJiKPBdoG/LQtwpHgWOSh2ElT9JnHjiiYwaNYrp06dv1X/XXXcxfvz4BJGZtW2KaLrGSNoMbCA7Q9od+HNdFxAR8almtv0CMCUixjZoF3ATMJ6swF0XEfdLqgAei4hhkiYCoyPiknybx4CbI6I6P3v5KXAC8B5ZgbsJGAhcHhGz8u1PBboBg4CZEXHVNr8Z0vqI6NFM/yRgEkDv3n1G/eDWO7a1yzav7+6w6oPUUbROqhyG9++5Vdvbb79N7969ee+997jyyiu59NJLOfzwwwG49957Wbx4MT/+8Y/JfgzqW79+PT16NPnfr2w4j7alreVx3HHHzYuI0Q3bt/Vwvc6tOOYwYF4j7V8BRgCHA72BFyU924L9dic727pa0kzgOmAcMBSoIpv1R36MkcBGYLGk2yJi6Xbk8bGImA5MBxh4wOC4ZUGxkxHbriuG11LueaTKoeacymb758+fz6ZNm6isrKSqqopFixYxZ84cunXr1uj61dXVVFY2v89y4DzalnLJo9jPGe1IY4AZEbE5IlaR3e/uyBZs/xEwO19eADyTT6hYQP07ic+JiLUR8SHwGrBfqyM3a8aGDRtYt27dx8tPPPEEw4YNY/bs2dx4443MmjWryUJk1tGV8s/JRTT+WaStxye2Vkv9Qtm1YHlTfDK2uIXszIeI2CKpMJ+NBcubKW2uZqxatYrTTz8dgNraWr7+9a9z0kknMXjwYDZu3Mi4ceOAbBLD7bffnjJUszanlL+gnwKul3RRRNwBIOlIsus8Z0mqAnoBY4HJ1C84NcC3JHUC+tMGJxfsvktnFreDDz1WV1dvc7iprWsrORxwwAHMnz9/q/YlS5YkiMasvJRsmC4/ezkdGJdP7V4ETAH+HXgVmE9WsK6KiJUNNn8OeJNs6O1m4KVSxQkg6SZJy4BukpZJmlLK45mZWX0lHbqKiLeAMxvpmpy/CtetIZv0UFfIzmlinz0Klqc01hcRd5PdabyufcI24rwK2OZsOzMzK40UExjMzMzq6VAX9SW9AOzWoPm8iFiQIh4zM8t0qGIUEUenjsHMzLbmYTozM0vOxcjMzJJzMTIzs+RcjMzMLDkXIzMzS87FyMzMknMxMjOz5FyMzMwsORcjMzNLzsXIzMySczEyM7PkXIzMzCw5FyMzM0vOxcjMzJJzMTIzs+RcjMzMLDkXIzMzS87FyMzMknMxMjOz5FyMzMwsORcjMzNLzsXIzMySczEyM7PkXIzMzCw5FyMzM0vOxcjMzJJzMTIzs+RcjMzMLDkXIzMzS65L6gDK1QebNlNxza9Th9FqVwyvZWIL8qi54ZSPl5cuXco3vvENVq5cSadOnZg0aRKXXXYZU6ZM4Y477qBPnz4AXH/99Zx88sk7PHYzaz9cjGy7denShVtuuYUjjjiCdevWMWrUKMaNGwfAt7/9ba688srEEZpZuSjpMJ2kfSTdJ+kNSa9JelzSkCbWrZC0sJTxNEXSKEkLJC2R9M+SlCKOctOvXz+OOOIIAPbYYw8OOeQQli9fnjgqMytHJStG+S/0mUB1RAyKiKHAd4G+pTpmK/wLMAk4MH+dlDac8lNTU8PLL7/M0UcfDcC0adM47LDDuOCCC3jvvfcSR2dmbZ0iojQ7lr4ATImIsQ3aBdwEjAcCuC4i7pdUATwWEcMkTQRGR8Ql+TaPATdHRLWk9cBPgROA98gK3E3AQODyiJiVb38q0A0YBMyMiKuaiLMf8HREHJy/PxuojIi/amTdSWRFi969+4z6wa13bPf3p63ouzus+qD49Yf377lV2wcffMBll13Gueeey9ixY3n33Xfp2bMnkrjrrrt45513uPrqq3dg1PWtX7+eHj16lGz/O4vzaFucR2kcd9xx8yJidMP2Ul4zGgbMa6T9K8AI4HCgN/CipGdbsN/uZGdbV0uaCVwHjAOGAlXArHy9EcBIYCOwWNJtEbG0kf31B5YVvF+Wt20lIqYD0wEGHjA4bllQ/pfcrhheS0vyqDmnst77TZs2MWHCBC6++GK+853vbLX+AQccwIQJE6isrNyqb0eprq4u6f53FufRtjiPnSvF1O4xwIyI2BwRq4BngCNbsP1HwOx8eQHwTERsypcrCtabExFrI+JD4DVgvyb219j1odKcLrYzEcGFF17IIYccUq8QrVix4uPlmTNnMmzYsBThmVkZKeWf9ouAMxppL2ZyQC31C2XXguVN8cnY4hayMx8iYoukwnw2FixvpulclwEDCt4PAN4qIsYO77nnnuOee+5h+PDhjBgxAsimcc+YMYNXXnkFSVRUVPCv//qvaQM1szavlMXoKeB6SRdFxB0Ako4ku85zlqQqoBcwFphM/YJTA3xLUieyIbOjShVkRKyQtE7SMcALwDeA20p1vPZkzJgxNHbN0Z8pMrOWKlkxioiQdDpwq6RrgA/JiszlQA9gPtlw2FURsTKfwFDnOeBNsqG3hcBLpYoz99fA3cDuwG/yV7N236Uziws+AFquqqurt7oOZGa2s5X0CnxEvAWc2UjX5PxVuG4N2aQH8mG4c5rYZ4+C5SmN9UXE3WTFpa59wjbinFt3bDMz2/l8bzozM0uu/Ocmt4CkF4DdGjSfFxELUsRjZmaZDlWMIuLo1DGYmdnWPExnZmbJuRiZmVlyLkZmZpaci5GZmSXnYmRmZsm5GJmZWXIuRmZmlpyLkZmZJediZGZmybkYmZlZci5GZmaWnIuRmZkl52JkZmbJuRiZmVlyLkZmZpaci5GZmSXnYmRmZsm5GJmZWXIuRmZmlpyLkZmZJediZGZmybkYmZlZci5GZmaWnIuRmZkl52JkZmbJuRiZmVlyLkZmZpaci5GZmSXnYmRmZsm5GJmZWXIuRmZmlpyLkZmZJediZGZmybkYmZlZcoqI1DGUJUnrgMWp49gBegNvpw6ildpDDuA82hrnURr7RUSfho1dUkTSTiyOiNGpg2gtSXPLPY/2kAM4j7bGeexcHqYzM7PkXIzMzCw5F6PtNz11ADtIe8ijPeQAzqOtcR47kScwmJlZcj4zMjOz5FyMzMwsORejFpJ0kqTFkpZIuiZ1PMWS9BlJT0v6naRFki7L23tJelLS7/Ove6WOtRiSOkt6WdJj+fuyy0PSnpIelPR6/u9ybJnm8e38/9RCSTMkdS2HPCTdJWm1pIUFbU3GLelv85/7xZK+mCbqrTWRx0/y/1evSpopac+CvjaZh4tRC0jqDPwUGA8MBc6WNDRtVEWrBa6IiEOAY4D/mcd+DTAnIg4E5uTvy8FlwO8K3pdjHv8EzI6Ig4HDyfIpqzwk9QcuBUZHxDCgM/A1yiOPu4GTGrQ1Gnf+s/I14NB8m/+V/z5oC+5m6zyeBIZFxGHAfwF/C207DxejljkKWBIRf4iIj4D7gNMSx1SUiFgRES/ly+vIfvH1J4u/Kl+tCvhykgBbQNIA4BTgZwXNZZWHpE8BY4E7ASLio4hYQ5nlkesC7C6pC9ANeIsyyCMingXebdDcVNynAfdFxMaIeBNYQvb7ILnG8oiIJyKiNn/7n8CAfLnN5uFi1DL9gaUF75flbWVFUgUwEngB6BsRKyArWMDeCUMr1q3AVcCWgrZyy+MA4E/Az/Phxp9J6k6Z5RERy4Gbgf8GVgBrI+IJyiyPAk3FXc4/+xcAv8mX22weLkYto0baympuvKQewEPA5RHxfup4WkrSBGB1RMxLHUsrdQGOAP4lIkYCG2ibQ1nNyq+pnAbsD+wLdJd0btqoSqIsf/YlfY9siP4XdU2NrNYm8nAxapllwGcK3g8gG5IoC5J2IStEv4iIX+XNqyT1y/v7AatTxVekzwGnSqohGyb9gqR7Kb88lgHLIuKF/P2DZMWp3PI4AXgzIv4UEZuAXwGfpfzyqNNU3GX3sy/pfGACcE588oHSNpuHi1HLvAgcKGl/SbuSXQiclTimokgS2fWJ30XEPxR0zQLOz5fPBx7Z2bG1RET8bUQMiIgKsu//UxFxLuWXx0pgqaSD8qbjgdcoszzIhueOkdQt/z92PNn1yHLLo05Tcc8CviZpN0n7AwcC/y9BfEWRdBJwNXBqRPy5oKvt5hERfrXgBZxMNjvlDeB7qeNpQdxjyE7HXwVeyV8nA58mmzX0+/xrr9SxtiCnSuCxfLns8gBGAHPzf5OHgb3KNI8fAa8DC4F7gN3KIQ9gBtl1rk1kZwwXNhc38L38534xMD51/NvIYwnZtaG6n/Xb23oevh2QmZkl52E6MzNLzsXIzMySczEyM7PkXIzMzCw5FyMzM0vOxcisAUmbJb1S8KrYjn18uVQ30ZW0r6QHS7HvZo45QtLJO/OY1rF0SR2AWRv0QUSMaOU+vgw8RvZB1qJI6hKf3NyySRHxFnDG9ofWMvkNUEcAo4HHd9ZxrWPxmZFZESSNkvSMpHmSfltwy5iLJL0oab6kh/I7EXwWOBX4SX5mNUhStaTR+Ta989sZIWmipF9KehR4QlL3/Pk0L+Y3UN3qrvCSKuqeXZNv/7CkRyW9KekSSd/Jt/1PSb3y9aol3SrpeWXPHToqb++Vb/9qvv5hefsUSdMlPQH8G/Bj4Kw8n7MkHZXv6+X860EF8fxK0mxlzwS6qSDukyS9lH+v5uRt28zXOojUn7r1y6+29gI288kn12cCuwDPA33y/rOAu/LlTxdsdx3wN/ny3cAZBX3VZM/8AegN1OTLE8k+Nd8rf389cG6+vCfZ3T66N4ivAlhYsP0SYA+gD7AWuDjv+0eyG+LWHf+OfHlswfa3AT/Ml78AvJIvTwHmAbsXHGdaQQyfArrkyycADxWs9wegJ9AV+CPZvdD6kN0RYP98vaLz9atjvDxMZ7a1esN0koYBw4Ans9uv0Zns9isAwyRdR/aLtAfw2+043pMRUfc8mhPJbgR7Zf6+KzCQ+g8SbOjpyJ5RtU7SWuDRvH0BcFjBejMge/6NpE8pe/rnGOAv8vanJH1aUs98/VkR8UETx+wJVEk6kOw2U7sU9M2JiLUAkl4D9iO71dGzkT1Dh1bma+2Qi5HZtglYFBHHNtJ3N/DliJgvaSLZ/fIaU8snw+JdG/RtaHCsv4iIxS2Ib2PB8paC91uo/zPe8N5fQfOPFNjQSF+da8mK4On5BI/qJuLZnMegRo4P25evtUO+ZmS2bYuBPpKOhexRHJIOzfv2AFYoezzHOQXbrMv76tQAo/Ll5iYf/Bb4m/wO2Ega2frwP3ZWvs8xZA/BWws8Sx63pErg7Wj8OVcN8+kJLM+XJxZx7P8APp/fKZq6a1mUNl8rIy5GZtsQ2SPmzwBulDSf7FrSZ/PuvyN7Yu6TZHeurnMfMDm/KD+I7Gmofy3pebJrRk25lmzI69V8ksK1OzCV9/Lj3052Z2fIrg2NlvQqcAOfPD6hoaeBoXUTGICbgKmSniMbtmxWRPwJmAT8Kv8e3p93lTJfKyO+a7dZByCpGrgyIuamjsWsMT4zMjOz5HxmZGZmyfnMyMzMknMxMjOz5FyMzMwsORcjMzNLzsXIzMyS+//0DbuLlP2FigAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 绘制重要的特征\n",
    "lgb.plot_importance(gbm,importance_type = \"split\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## xgboost模型\n",
    "[XGBoost基础介绍](https://mp.weixin.qq.com/s/AAKPSIHk1iUqCeUibrORqQ)  \n",
    "[XGBoost参数介绍](https://xgboost.readthedocs.io/en/latest/parameter.html)  \n",
    "[XGboost参数调优方法](https://blog.csdn.net/han_xiaoyang/article/details/52665396)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1_score: 95.56%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEWCAYAAABliCz2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAWFklEQVR4nO3de7hddX3n8feHmAolXEQIKhi5xNYqN5HBOlIMHaQq+KitomhVELQ+FlFHqVWLgK3VKiidaWdUUMcroyMWMiOD4mikVq0CcqsUbxzlJjHRoIlRkvCdP/ZCD8ecnB04++bv/Xqe82Tttfbe67MXnM9Z+7fWXjtVhSTpN982ow4gSRoOC1+SGmHhS1IjLHxJaoSFL0mNsPAlqREWvjRDktcnOW/UOaT5Fs/D13xKMgXsDmyaNvt3qurW+/icJ1XVZ+9busmT5AxgaVX96aizaPK5h69BeGpVLZr2c6/Lfj4kud8o139vTWpujS8LX0ORZKck701yW5JbkvxNkgXdsn2TfC7J6iSrknwkyc7dsg8BS4D/nWRtkr9IsizJzTOefyrJkd30GUk+keTDSX4CHL+l9W8m6xlJPtxN75WkkpyQ5KYkP07y0iT/Ick1SdYk+Ydpjz0+yb8k+a9J7kjy70n+07TlD0myPMmPknw7yYtnrHd67pcCrwee3b32q7v7nZDk+iQ/TfLdJH827TmWJbk5yauTrOxe7wnTlm+X5Owk3+vyfTHJdt2y30/ype41XZ1k2b34T60xZuFrWD4AbASWAo8GjgJO6pYFeAvwEOD3gIcCZwBU1fOB7/Ordw1v63N9TwM+AewMfGSO9ffjscDDgWcD5wBvAI4EHgUcm+QJM+77XWBX4HTgk0l26ZadD9zcvdZnAn87/Q/CjNzvBf4W+Fj32g/s7rMSOAbYETgBeGeSg6c9x4OAnYA9gBOBf0zygG7ZWcBjgP8I7AL8BXBXkj2ATwF/081/DXBBkt22YhtpzFn4GoQLu73ENUkuTLI78GTglVW1rqpWAu8EngNQVd+uqkur6hdV9UPgHcATZn/6vny5qi6sqrvoFeOs6+/TX1fVz6vqM8A64PyqWllVtwD/TO+PyN1WAudU1Yaq+hhwA3B0kocChwGv7Z7rKuA84Pmby11V6zcXpKo+VVXfqZ4vAJ8B/mDaXTYAb+rWfzGwFvjdJNsALwJeUVW3VNWmqvpSVf0C+FPg4qq6uFv3pcDlwFO2YhtpzDlGqEF4+vQDrEkOBRYCtyW5e/Y2wE3d8sXAf6FXWjt0y358HzPcNG36YVtaf59unza9fjO3F027fUvd82yI79Hbo38I8KOq+umMZYfMknuzkjyZ3juH36H3On4buHbaXVZX1cZpt3/W5dsV2Bb4zmae9mHAs5I8ddq8hcDn58qjyWHhaxhuAn4B7DqjiO72FqCAA6pqdZKnA/8wbfnMU8nW0Ss5ALqx+JlDD9MfM9f659seSTKt9JcAy4FbgV2S7DCt9JcAt0x77MzXeo/bSe4PXAC8ALioqjYkuZDesNhcVgE/B/YFrp6x7CbgQ1X14l97lH5jOKSjgauq2+gNO5ydZMck23QHau8ettmB3rDDmm4s+dQZT3E7sM+0298Etk1ydJKFwF8B978P659vi4FTkixM8ix6xyUurqqbgC8Bb0mybZID6I2xf2QLz3U7sFc3HAPwW/Re6w+Bjd3e/lH9hOqGt94HvKM7eLwgyeO6PyIfBp6a5I+6+dt2B4D33PqXr3Fl4WtYXkCvrL5Bb7jmE8CDu2VnAgcDd9A7cPjJGY99C/BX3TGB11TVHcDL6I1/30Jvj/9mtmxL659v/0rvAO8q4M3AM6tqdbfsOGAvenv7/wSc3o2Xz+Z/df+uTnJl987gFODj9F7Hc+m9e+jXa+gN/3wN+BHwd8A23R+jp9E7K+iH9Pb4T8WO+I3iB6+keZTkeHofEjts1FmkmfzrLUmNsPAlqREO6UhSI9zDl6RGjO15+DvvvHMtXbp01DG2yrp169h+++1HHWOrTFrmScsLZh4WM/dcccUVq6pqs5fEGNvC33333bn88stHHWOrrFixgmXLlo06xlaZtMyTlhfMPCxm7knyvdmWOaQjSY2w8CWpERa+JDXCwpekRlj4ktQIC1+SGmHhS1IjLHxJaoSFL0mNsPAlqREWviQ1wsKXpEZY+JLUCAtfkhph4UtSIyx8SWqEhS9JjbDwJakRFr4kNcLCl6RGWPiS1AgLX5IaYeFLUiMsfElqhIUvSY2w8CWpERa+JDXCwpekRlj4ktQIC1+SGmHhS1IjLHxJaoSFL0mNsPAlqREWviQ1wsKXpEZY+JLUCAtfkhph4UtSIyx8SWqEhS9JjbDwJakRFr4kNcLCl6RGWPiS1AgLX5IaYeFLUiMsfElqhIUvSY2w8CWpERa+JDXCwpekRlj4ktQIC1+SGmHhS1IjLHxJaoSFL0mNsPAlqREWviQ1wsKXpEZY+JLUCAtfkhph4UtSIyx8SWpEqmrUGTZryT5La5tj/37UMbbKq/ffyNnX3m/UMbbKpGWetLxg5mEZt8xTbz16zvusWLGCZcuWzet6k1xRVYdsbpl7+JLUCAtfkhph4UvSiNx5550ceuihHHjggTzqUY/i9NNPv8fys846iySsWrVqXtY3sMJPckqS65NUkmu6ny8lOXBQ65SkSbJw4UI+97nPcfXVV3PVVVdxySWX8JWvfAWAm266iUsvvZQlS5bM2/oGuYf/MuApwOOBJ1TVAcBfA+8Z4DolaWIkYdGiRQBs2LCBDRs2kASAV73qVbztbW/75e35MJDCT/IuYB9gOfDYqvpxt+grwJ6DWKckTaJNmzZx0EEHsXjxYp74xCfy2Mc+luXLl7PHHntw4IHzOyAysNMyk0wBh1TVqmnzXgM8oqpOmuUxLwFeArDrrrs95o3nnDuQbIOy+3Zw+/pRp9g6k5Z50vKCmYdl3DLvv8dOc95n7dq1v9zDX7t2Laeddhonn3wyZ511Fm9/+9tZtGgRz3nOc3j3u9/NTjvN/XwARxxxxKynZQ7tpNUkRwAnAofNdp+qeg/dkM+SfZbWOJ1T249xOw+4H5OWedLygpmHZdwyTz1v2Zz3mXke/hVXXMGtt97K6tWrOfnkkwFYtWoVL3/5y/nqV7/Kgx70oPuUaShn6SQ5ADgPeFpVrR7GOiVp3K1Zs4Y1a9YAsH79ej772c/y6Ec/mpUrVzI1NcXU1BR77rknV1555X0uexjCHn6SJcAngedX1TcHvT5JmhSrV6/miCOOYNOmTdx1110ce+yxHHPMMQNb3zDe/7wReCDw37qjzRtnG1+SpJbsu+++fP3rX9/ifaampuZtfQMr/Kraq5s8qfuRJI2Qn7SVpEaMzyHtGbZbuIAb+rja3DhZsWJFX0fmx8mkZZ60vGDmYZnEzMPmHr4kNcLCl6RGWPiS1AgLX5IaYeFLUiMsfElqhIUvSY2w8CWpERa+JDWir8JPsm+S+3fTy7rvq915oMkkSfOq3z38C4BNSZYC7wX2Bj46sFSSpHnXb+HfVVUbgWcA51TVq4AHDy6WJGm+9Vv4G5IcB7wQ+D/dvIWDiSRJGoR+C/8E4HHAm6vqxiR7Ax8eXCxJ0nzr6/LIVfWNJK8FlnS3bwTeOshgkqT51e9ZOk8FrgIu6W4flGT5AHNJkuZZv0M6ZwCHAmsAquoqemfqSJImRL+Fv7Gq7pgxr+Y7jCRpcPr9isPrkjwXWJDk4cApwJcGF0uSNN/63cN/OfAo4Bf0PnB1B/DKAWWSJA3AnHv4SRYAy6vqSOANg48kSRqEOffwq2oT8LMkOw0hjyRpQPodw/85cG2SS4F1d8+sqlMGkkqSNO/6LfxPdT+SpAnV7ydtPzDoIJKkweqr8JPcyGbOu6+qfeY9kSRpIPod0jlk2vS2wLOAXeY/jiRpUPo6D7+qVk/7uaWqzgH+cLDRJEnzqd8hnYOn3dyG3h7/DgNJJEkaiH6HdM6eNr0RuBE4dv7jSJIGpd/CP7Gqvjt9RvclKJKkCdHvtXQ+0ec8SdKY2uIefpJH0Lto2k5J/njaoh3pna0jSZoQcw3p/C5wDLAz8NRp838KvHhAmSRJA7DFwq+qi4CLkjyuqr48pEySpAHo96Dt15P8Ob3hnV8O5VTViwaSSpI07/o9aPsh4EHAHwFfAPakN6wjSZoQ/Rb+0qo6DVjXXUjtaGD/wcWSJM23fgt/Q/fvmiT7ATsBew0kkSRpIPodw39PkgcApwHLgUXAGweWSpI07/q9Hv553eQXAC+JLEkTqK8hnSS7J3lvkv/b3X5kkhMHG02SNJ/6HcP/H8CngYd0t78JvHIAeSRJA9Jv4e9aVR8H7gKoqo3ApoGlkiTNu34Lf12SB9J9zWGS3wfuGFgqSdK86/csnf9M7+ycfZP8C7Ab8MyBpZIkzbu5rpa5pKq+X1VXJnkCvYupBbihqjZs6bGSpPEy15DOhdOmP1ZV/1ZV11n2kjR55ir8TJv2/HtJmmBzFX7NMi1JmjBzHbQ9MMlP6O3pb9dN092uqtpxoOkkSfNmri9AWTCsIJKkwer3PHxJ0oSz8CWpERa+JDXCwpekRlj4ktSIfq+lM3TrN2xir7/81KhjbJVX77+R4808q6m3Hj2U9UjaPPfwJakRFr4kNcLC19h50YtexOLFi9lvv/1+Oe+0007jgAMO4KSTTuKoo47i1ltvHWFCaTINrPCTnJLk+iQ/TnJNkquSXJ7ksEGtU78Zjj/+eC655JJ7zDv11FO55pprOO+88zjmmGN405veNKJ00uQa5EHblwFPBn4IrKuqSnIA8HHgEQNcrybc4YcfztTU1D3m7bjjry7btG7dOpIgaesMpPCTvIve5ZSXA++rqnd2i7bHq27qXnrDG97Aueeey+LFi/n85z8/6jjSxEnVYPo3yRRwSFWtSvIM4C3AYuDoqvryLI95CfASgF133e0xbzzn3IFkG5Tdt4Pb1486xdYZZub999ip7/v+4Ac/4HWvex3vf//77zF/7dq1XHTRRdx5552ccMIJ8x1xINauXcuiRYtGHWOrmHk4BpH5iCOOuKKqDtncsqEU/rR5hwNvrKoj53r8kn2W1jbH/v1Asg3Kq/ffyNnXju1HGzZrmJm35jz8qakpjjnmGK677rp7zF+xYgV77703Rx999K8tG1crVqxg2bJlo46xVcw8HIPInGTWwh/qWTpVdRm9L0LfdZjr1eT71re+9cvp5cuX84hHeBhI2loD37VLshT4TnfQ9mDgt4DVg16vJtdxxx3HihUrWLVqFXvuuSdnnnkmF198MTfccAPr16/nkY98JO9617tGHVOaOMN4L/8nwAuSbADWA8+uQY0j6TfC+eef/2vzTjzxRGAy37ZL42JghV9Ve3WTf9f9SJJGyE/aSlIjLHxJasTYnkO43cIF3DBhl9NdsWIFU89bNuoYW2USM0u6d9zDl6RGWPiS1AgLX5IaYeFLUiMsfElqhIUvSY2w8CWpERa+JDXCwpekRlj4ktQIC1+SGmHhS1IjLHxJaoSFL0mNsPAlqREWviQ1wsKXpEZY+JLUCAtfkhph4UtSIyx8SWqEhS9JjbDwJakRFr4kNcLCl6RGWPiS1AgLX5IaYeFLUiMsfElqhIUvSY2w8CWpERa+JDXCwpekRlj4ktQIC1+SGmHhS1IjLHxJaoSFL0mNsPAlqREWviQ1wsKXpEZY+JLUCAtfkhph4UtSIyx8SWqEhS9JjbDwJakRFr4kNcLCl6RGWPiS1AgLX5IaYeFLUiMsfElqhIUvSY2w8CWpERa+JDXCwpekRlj4ktQIC1+SGmHhS1IjLHxJaoSFL0mNsPAlqREWviQ1IlU16gybleSnwA2jzrGVdgVWjTrEVpq0zJOWF8w8LGbueVhV7ba5Bfeb5xXNpxuq6pBRh9gaSS4382BNWl4w87CYeW4O6UhSIyx8SWrEOBf+e0Yd4F4w8+BNWl4w87CYeQ5je9BWkjS/xnkPX5I0jyx8SWrEWBZ+kicluSHJt5P85ajz9CPJVJJrk1yV5PJR55kpyfuSrExy3bR5uyS5NMm3un8fMMqMM82S+Ywkt3Tb+aokTxllxpmSPDTJ55Ncn+Tfkryimz+W23oLecd2OyfZNslXk1zdZT6zmz+W2xi2mHmo23nsxvCTLAC+CTwRuBn4GnBcVX1jpMHmkGQKOKSqxvKDH0kOB9YCH6yq/bp5bwN+VFVv7f6wPqCqXjvKnNPNkvkMYG1VnTXKbLNJ8mDgwVV1ZZIdgCuApwPHM4bbegt5j2VMt3OSANtX1dokC4EvAq8A/pgx3MawxcxPYojbeRz38A8Fvl1V362qO4H/CTxtxJkmXlVdBvxoxuynAR/opj9A7xd9bMySeaxV1W1VdWU3/VPgemAPxnRbbyHv2Kqetd3Nhd1PMabbGLaYeajGsfD3AG6advtmxvx/wE4Bn0lyRZKXjDpMn3avqtug94sPLB5xnn6dnOSabshnbN62z5RkL+DRwL8yAdt6Rl4Y4+2cZEGSq4CVwKVVNfbbeJbMMMTtPI6Fn83MG69xp817fFUdDDwZ+PNuOELz778D+wIHAbcBZ480zSySLAIuAF5ZVT8ZdZ65bCbvWG/nqtpUVQcBewKHJtlvxJHmNEvmoW7ncSz8m4GHTru9J3DriLL0rapu7f5dCfwTvaGpcXd7N4Z791juyhHnmVNV3d794twFnMsYbudujPYC4CNV9clu9thu683lnYTtDFBVa4AV9MbCx3YbTzc987C38zgW/teAhyfZO8lvAc8Blo840xYl2b474EWS7YGjgOu2/KixsBx4YTf9QuCiEWbpy92/0J1nMGbbuTs4917g+qp6x7RFY7mtZ8s7zts5yW5Jdu6mtwOOBP6dMd3GMHvmYW/nsTtLB6A7NekcYAHwvqp682gTbVmSfejt1UPvCqQfHbfMSc4HltG7HOvtwOnAhcDHgSXA94FnVdXYHCSdJfMyem9/C5gC/uzucdtxkOQw4J+Ba4G7utmvpzcuPnbbegt5j2NMt3OSA+gdlF1Ab6f141X1piQPZAy3MWwx84cY4nYey8KXJM2/cRzSkSQNgIUvSY2w8CWpERa+JDXCwpekRozzl5hLA5FkE73TEO/29KqaGlEcaWg8LVPNSbK2qhYNcX33q6qNw1qfNBuHdKQZkjw4yWXd9cmvS/IH3fwnJbmyu6b5/+vm7ZLkwu7iV1/pPmBz93XO35PkM8AHu09aXpDka93P40f4EtUoh3TUou26qxYC3FhVz5ix/LnAp6vqzd33M/x2kt3oXevk8Kq6Mcku3X3PBL5eVU9P8ofAB+l9chLgMcBhVbU+yUeBd1bVF5MsAT4N/N7AXqG0GRa+WrS+u2rhbL4GvK+7qNiFVXVVkmXAZVV1I8C0j+wfBvxJN+9zSR6YZKdu2fKqWt9NHwk8snfpGgB2TLJDdw16aSgsfGmGqrqsu7z10cCHkrwdWMPmL9O9pct5r5s2bxvgcdP+AEhD5xi+NEOShwErq+pceleSPBj4MvCEJHt397l7SOcy4HndvGXAqlmuf/8Z4ORp6zhoQPGlWbmHL/26ZcCpSTbQ+07dF1TVD7tvMvtkkm3oXWv9icAZwPuTXAP8jF9dnnemU4B/7O53P3p/KF460FchzeBpmZLUCId0JKkRFr4kNcLCl6RGWPiS1AgLX5IaYeFLUiMsfElqxP8HeF5Uwdd3eicAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "import xgboost as xgb\n",
    "from xgboost import plot_importance\n",
    "from matplotlib import pyplot as plt\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import f1_score   # 准确率\n",
    "# 加载样本数据集\n",
    "iris = load_iris()\n",
    "X,y = iris.data,iris.target\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1234565) # 数据集分割\n",
    "# 算法参数\n",
    "params = {\n",
    "    'booster': 'gbtree',\n",
    "    'objective': 'multi:softmax',\n",
    "    'eval_metric':'mlogloss',\n",
    "    'num_class': 3,\n",
    "    'gamma': 0.1,\n",
    "    'max_depth': 6,\n",
    "    'lambda': 2,\n",
    "    'subsample': 0.7,\n",
    "    'colsample_bytree': 0.75,\n",
    "    'min_child_weight': 3,\n",
    "    'eta': 0.1,\n",
    "    'seed': 1,\n",
    "    'nthread': 4,\n",
    "}\n",
    "\n",
    "# plst = params.items()\n",
    "\n",
    "train_data = xgb.DMatrix(X_train, y_train) # 生成数据集格式\n",
    "num_rounds = 500\n",
    "model = xgb.train(params, train_data) # xgboost模型训练\n",
    "\n",
    "# 对测试集进行预测\n",
    "dtest = xgb.DMatrix(X_test)\n",
    "y_pred = model.predict(dtest)\n",
    "\n",
    "# 计算准确率\n",
    "F1_score = f1_score(y_test,y_pred,average='macro')\n",
    "print(\"F1_score: %.2f%%\" % (F1_score*100.0))\n",
    "\n",
    "# 显示重要特征\n",
    "plot_importance(model)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://blog.csdn.net/han_xiaoyang/article/details/52665396\n",
    "https://www.cnblogs.com/TimVerion/p/11436001.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 交叉验证"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "交叉验证是验证分类器性能的一种统计分析方法，其基本思想在某种意义下将原始数据进行分组，一部分作为训练集，另一部分作为验证集。首先是用训练集对分类器进行训练，再利用验证集来测试所得到的的模型，以此来作为评价分类器的性能指标。常用的交叉验证方法包括简单交叉验证、K折交叉验证、留一法交叉验证和留P法交叉验证"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.简单交叉验证(cross validation)  \n",
    "简单交叉验证是将原始数据分为两组，一组作为训练集，另一组作为验证集，利用训练集训练分类器，然后利用验证集验证模型，将最后的分类准确率作为此分类器的性能指标。通常是划分30%的数据作为测试数据\n",
    "2.K折交叉验证(K-Fold cross validation)  \n",
    "K折交叉验证是将原始数据分为K组，然后将每个子集数据分别做一次验证集，其余的K-1组子集作为训练集，这样就会得到K个模型，将K个模型最终的验证集的分类准确率取平均值，作为K折交叉验证分类器的性能指标。通常设置为K为5或者10.  \n",
    "3.留一法交叉验证(Leave-One-Out Cross Validation，LOO-CV)\n",
    "留一法交叉验证是指每个训练集由除一个样本之外的其余样本组成，留下的一个样本组成检验集。这样对于N个样本的数据集，可以组成N个不同的训练集和N个不同的验证集，因此该方法会得到N个模型，用N个模型最终的验证集的分类准确率的平均是作为分类器的性能指标。  \n",
    "4.留P法交叉验证  \n",
    "该方法与留一法类似，是从完整数据集中删除P个样本，产生所有可能的训练集和验证集。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "交叉验证示例代码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.简单交叉验证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import datasets\n",
    "#数据集导入\n",
    "iris=datasets.load_iris()\n",
    "feature=iris.feature_names\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.4,random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.K折交叉验证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold\n",
    "folds = KFold(n_splits=10, shuffle=is_shuffle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.留一法交叉验证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import LeaveOneOut\n",
    "loo=LeaveOneOut()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.留P法交叉验证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import LeavePOut\n",
    "lpo=LeavePOut(p=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另外还有一些其他交叉验证的分割方法，如基于类标签，具有分层的交叉验证。这一类交叉验证方法主要用于解决样本不平衡的问题。  \n",
    "这种情况下常用StratifiedKFold和StratifiedShuffleSplit的分层抽样方法，可以确保相应的类别频率在每个训练和验证的(fold)中得以保留。  \n",
    "StratifiedKFold：是K-fold的变种，会返回stratified(分层)的折叠：每个小集合中的各个类别的样本比例大致和完整数据集相同。  \n",
    "StratifiedShuffleSplit:是ShuffleSplit的一种变种，会返回直接的划分，比如创建一个划分，但是划分中的每个类的比例和完整数据集中的相同。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 模型调参\n",
    "调参就是对模型的参数进行调整，找到使模型最优的超参数，调参的目标就是尽可能达到整体模型的最优"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.网格搜索  \n",
    "网格搜索就是一种穷举搜索，在所有候选的参数选择中通过循环遍历去在所有候选参数中寻找表现最好的结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.学习曲线  \n",
    "学习曲线是在训练集大小不同时通过绘制模型训练集和交叉验证集上的准确率来观察模型在新数据上的表现，进而来判断模型是否方差偏高或偏差过高，以及增大训练集是否可以减小过拟合。  \n",
    "\n",
    "<img src=\"https://img-blog.csdn.net/20180207143722125?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvZGF0b3V0b25nXw==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast\" alt=\"img\" style=\"zoom:50%;\" />\n",
    "1、当训练集和测试集的误差收敛但却很高时，为高偏差。   \n",
    "\n",
    "左上角的偏差很高，训练集和验证集的准确率都很低，很可能是欠拟合。   \n",
    "我们可以增加模型参数，比如，构建更多的特征，减小正则项。   \n",
    "此时通过增加数据量是不起作用的。\n",
    "  \n",
    "2、当训练集和测试集的误差之间有大的差距时，为高方差。   \n",
    "\n",
    "当训练集的准确率比其他独立数据集上的测试结果的准确率要高时，一般都是过拟合。   \n",
    "右上角方差很高，训练集和验证集的准确率相差太多，应该是过拟合。   \n",
    "我们可以增大训练集，降低模型复杂度，增大正则项，或者通过特征选择减少特征数。  \n",
    "理想情况是是找到偏差和方差都很小的情况，即收敛且误差较小。"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaoAAAEXCAYAAAD82wBdAAAgAElEQVR4AeydB5wUVfLHf5PD5sAG2EDOIEhQFANBQUWSKKggoHfmeGbPeKbTv2cOp545ggKiKCBRgohEyQISloVlc97JM/9PvdmenbxpZmM1jt39cn+7t6tfVb33ZA6HwwHemAATYAJMgAm0UALyFtoubhYTYAJMgAkwAUGABRU/CEyACTABJtCiCbCgatG3hxvHBJgAE2ACLKj4GWACTIAJMIEWTSCkguqG62ciNbUTzjjjjIAXfeftN6FPn74YNGgQdu/c4kq3Ytn36Ne/n4h78bmnXOF8wASYABNgAu2bQEgF1d+uvx4/L/0uINHlPy7CX0eysH//Pnzw3pv42213ibQ2mw133PMglv34A3bt2okvv1mEgwf2BCyHI5gAE2ACTKD9EAipoBoxcjRi4xIC0lv0/feYO2uGiB8yfCQqSkuRl3sK27dsRJfMDKRndoVKpcaMaZOxcMG3AcvhCCbABJgAE2g/BEIqqGrDdvJUHjI6d3Yl65iaipyTWTiZnY20tBRXeEZ6Ok6eOuU65wMmwASYABNovwSUTX3pjRlfLJPJmrq5XB8TYAJMgAnUkwCZc+Ty0PWDQldSHS6kU8ckZB075kp5KicHHdMykZaejuzs067w41lZSO/UyXXufkCCLtS/Jx57sN5l1iVPsDSB4rzDvc/p2t3D3I+940LFybuOupRblzzB0gSK8xfuHRbs3DuuLtdSlzQNKbcueYKl8RdXlzDvNLWd1+X6a0vjXUdt6Sn+yccfgsNuD/q3GSwNxdn95PcX7h0W7Nw7ri7XUpc09S23rumDpfMXV5cw7zTu56EUUNK7v0kF1ZSJE/Hx5/NE3Vs3r0dkTAw6JKVi0JAROHrsOE4cPwKL2YR5CxZj8tQrpDaGfT92zJh611GXPMHSBIrzDvc+p4a6h7kf1/si6pihIXXUJU+wNIHi/IV7h9V2XsfLrlcy7zrrkrkueYKl8RdXlzDvNN7ndWl7fdM0pI4xo0cDtWhRgqWhOH9aGH/h3mG1ndf3+uuS3rvO2vLUNX2wdP7i6hLmncb7vLa21zdeFsq5/q6eMQ0bft2CwtM5SE5Lx+OP3AuLxSLadOMtTg+/22/9O1au2QCtRoNPP3oPAwcPF/E//7QYd9//iPgCmn3tVXj4UV8XdXro6MuEt8AE5s65Bh99/GXgBO08hvnU/gAwo+CMrp97LT748HO/QjB4zuaNpVen35/dfzi80tur09He/Udl2lxhDvToqRTv/VD2rEJqo/rq69o99d58+32/d+viSydh36WT/MZxYN0JDBowsO6J22FK5lP7TWdGwRnVh49fwUDf2gGEgHe4e34SDlarU0h4CwfPc6fgsFkBBwmXagHjfVVSZ5XySpsURufu4dI5xYuf3K2z6xZmt0olhXYf0h5VaJvmWxr3qHyZcIh/AnaHHTaHTfy10TH9HPTP4XB9CcvgdM6R1EN07n4sSq7+y5VD7hMnpfXfAg5tSQSk3gC9fOnYXSC4x0kvdSmdtCcBsXDhD/hk4WqR99pJo3HZhAluPQmncBDp6bGrFhCiLtS81N0FgTsfyhdsE/kkgSD5lMkAuaJacNBxdbgQJG5p6VyKC1ZHY+N++OFHfLV4FX7/5TXYbNaQOlOwoGrs3eH8DSZAQsMpRGxCiLgEigi3VQsXiL3VboMddtjsVljsVrEnQWSz22BxWEScxW6D1W4FiSTaZA76I3b+9VKYvFowuTdYCncXYFK8ezmO6peAlI6EHgk2qUy5zGnulfYKmRzy6rpJyNEmlymgkI6rPaIkYamQKSDllbmVReU48zr3rjjIIcVJ1+gpaEU2QMovXpbOi6BW00b5pPrdy1DKVVDQGzBMm7tgcBca7uH04g4mNKS0FtHDoA8QwHlcIzBsFs8yqm+H66qkc8rrfiwloHBxm2TA8mVL8J/XsuEovNEZHf8eHrovDZdMmOASApSWypF+TSEcpLY2956E1HMvnIAt/+8osmtgs1lCKqhCqvprblhcP7DhlxUYecFFIUNBL2b6R0LBIfVMqgWMgwSH8MJ0emaRoLE4JCEiCRWLyE/CxUoCxeEQgsZKvR23TXrhS4KAorwFhfNl7nzZ0ktdKaNczl6QEipolBrXuVvRHoebf9mIsy441yMsFCcSJyqLjmmj9tM/EeYUs0TMeW6riSMmtIn01cdSPhFRXaYzFZVPXQJPPpRXEjZSHmnvLrClMErvzpxe/D8v2ogfF+xHWVkeZlx3AS6ZOgZauQ4ahRYamRYqqCCXKSGXqaCEGnKHAjabTPRQqNmBhIbdBtDPXfBIgkFqD51XX3qtQkMSBPTdIY7lToEiCQaVsjpcAyicMlmqJujeagNMZsBqAsxm589iBowmYP7CNXAUviiYFehWIr7oRrz35f2IjLoMdgfZzp3tJ470I5sNhUnH9BhIcSKty6ZDH2LOOGt1Hlu1zUfKT9wor5RP4ijKo7zV6aU6pXxSHvd6hS2puj5KR3WJx0kKc6uH8tHTSu2yOJx7egalekV76N46gLxjq6EvJT71AB70bnhGsqDy5NGqz4SgsFpgshpdqi67EDBSz8W5twm1mLNnQnmol2J1WIUjC/VOSIhQGuqdUD/G9dXtp4ciXpD0sFf3OOjlR0JE+lqnr3760blCpoRaJodeoQv4Um2tN0C6XtH+ahbh65PUUJJeGtLLSLxcvF+aVoB8miw2gOwW9LKiPQkPi9UhzjetXIt5b2thLX0PRboVeO9fh1FcsBFDRp8Lu70MFoezh0s9OnpByeVOsalTaqCR66BXaqBVaqFXaqGUKaGSKaFUqKBWqiBXOQWHJDToxUaqNJMJsJqdbTObnOdGEhAmgAQECQtxbnEKEDqncBImhuowym+1AEa3NCYLYLY4y6bjKitgtDrLo3wKC6AyAKhyQF5ZzaH6Ba2SA0o5YIsAoJfBqgfMKsCQA+hrsIuj/FPAp/OdnVZ3gUmdWDonwSn1smivqBasdOwe7n5OjOgn8rrlF2VSuAJOnm5lUHqqTyqf2i+FSWWTUJeOpbaKeqS8buVROim/lEbspTKq1Y2UTrrWZ18AsnZ6AQrhKav+QgizOYqir/cKcxnyqwqQa8wXTfD+WqaXKIXRP7nD2QOhFw6ppugfCRI6FgJGfDNKYeH5OmoOTi2tTncB4/EFTF+w9JVaLUxsNudLncLo5S6M6dUChl74ddncX2Jyucxl16AXDZVfVirHMw+/jIKdb7l6WfS8RPa4HWMveRwWiwwmqwxmi1wIBYtZBotVLsKMZofY201AhcWBXKsMNqscDqscaoscCpMMKosCqgo5FEYZHDYZLHaI724lvXRJmutl4mfRATYVoFMBajWgVQJ6JaBRAUo1oFY5jylOQ/EUXh1H55SP9ipKW52G8mirz0W4xnmu0QBqDaCS8msApTKwLef775fg+RezYct3qv7kie/hkQfTMHHihLrcgjafhlV/bf4WN+wCqyyVKDQWIafytOj5KBVKxKlj2lxPpWF0wpdLCBVSmTj9NGrUOahW91R7ZklChQQMpRUCx0J7h1ClUDkkQPxtFEeb+LJ1fYXTB4bzC5ZezmqN89w9P9VZUixHabESZUUKlJbKUVaiQEkZHStQWiFHcakSJ8oVUJUqoTulgNUigyzKjlKlGtSJcN/MFQpkZauhVtmhIAGgsiMqwgFNnANKlQMKlQNKtQMalUMIBrXKAZWajh1QapzHlE6uskKptkKhtECmskCutEChcKqyRE9UOLgooJFroZVroVJooZPpQLYypUMJGfXOSN2oUAg1k8SH9pIaTLTbTXCLODqvDnOduzk2SNdKtiwDech53RP38/PPmwBD1RIsWnqfyDblktG44PwJKCl23kcpbV32Ur3ue/d8Uri439IzUr2XnhlpL6WVzK9SeLVp0vWMSeGSepTySWG17d3zuOrzOrj88ssgk/2IL7+7H7//4nYjvNI19JR7VA0l1wz5zDYTio0lOFl1ClUWA1RyJaKUER6G73DZYJrhckNeJb0MNq7diOEjSZ1VLWzcBAypxEiNREKF7C304pcEjN3uEKqzujZKUt0oFDUCRlKV1OUPn1R1JSVylJHQKVagtESOkhKnwCkrU6C4XCmEz4kKOVQlSmhPKWCzyiCLsaMyyQpdlA0p0TbERdkRG21DbJQVUbE2xMbaERNnQ3ScDbHxVugjHOJlv3bpGnz5ug3W0luE6i9JfRjTb1VgxNgLXS+0ul57XdJJL2ZKS8ckUewyq/NHamgZWUccoA6XjdRScKqjIlRkM9NBq9AgUqWFSqaBWqWERq6ESqmESq6C6DVWq7KoZMHd7UVPL2b3l7M4ri1eerHLyA78M0ZecLEQgs62S9dQ+95dcNZcu2c+7zLpWaUt0F5KT8+vZzqnwAgc70wv5aN0pA6mTdit3PZUN3GiNO57Z2rP/w87SwmrlZ0pPKm08TOL3YIKUzlyKnNQbCkTVxul0qODNr6NX3njL4+EDdk4jAagtAwor3Dgr2N2RCT6fvFJf4CSgHG+3JxqMoUSUClk0Elft/VsmiR0RE+n0LOnU16qQGG5AmVlSmSVK6AuVkB72lPoREXakBhjQ0KUDdExNiQlWNGtmwkxMU6hExNvQ0ycFZGRDvFSpuZJgphePFIvzl+ziY1GI8PFk0ZBG7EGy5fcgoTyXFw9ezzGTxzl9GjzerETI4kgCRB6cYmXvZ9jqtM7TsLong+iS6AGQL/Am/D4dBhhtlcIO2qVwwYjnKptykXqbY1cDZ1CiwilXtjNNAoN5CTI5EooxY+cQhqu1larZaIHGbiVLSVGIh269khCL9heut+hq5U8eCUXpVCWGqaySE3QiprbYAp0jWXmMhRWFSDHmCfKiVBqoVPoGlxmW89IL2YystOvtByoKHPA7Db4cPOatVi9cod4HV50yWCMumRUg5CYzTKUltBPibJCBYpL5KgoVaCoVAESOsUVJIiUOF6ugLbEU+hUJFsRHWlDhxgb4knoRNPeiijq4cQ4EBtf3dOpFjref/D0chACSPT0nE4QgexUKoVTPShsPWSLUTttMMLQroBQvZGNyLuOBkFpYZloyIIVVpjFMAbnEAZy9iFHH/rbIvsbCS2dQgMtCTRVBDRyDVQKEmQqlzALp4t+C0MW0uYoFIqQz0zBgiqkt6hxhUl2p1OVOWJ8ENmdopWRbHfyg5W8uqi3VFUJlJUDlVXSNz71fpwvabKD0LZm6Rp8/JIN9rJbxLk8+h1cf78CI0aPFkKnuFiJ8kIFSkrlKC9xCp2KMgWKyp1CJ6tCAT2p13IUTvtSrB0VSVbERNiQFGtDfLQN0VG0tyIyxoaYWAfi4m2IjLUhLt6KiAhyHXe2xd//SQAJO5YkgKoFkr88GpUMKnIEIGeCascBcgJwF0CSx5a/ujjMScA5lMICGp9HXq7UU6NNctmXHJLIq5E+EMlFX6/UQ61QCfWiWqGGigx3vPkQYEHVBntU5EpebCrBqaqcgHYnnychSEBbtFGRDp1cmUkwlZYCZWXkZu+EQC9zrUYmvLf8YSE37LtvfhUlfzg92sQYGMMYWCPuhdbwHzji7KjoYEWs1NMhoRNNajYr9CRoYuyipxMVS/vahY57G6iNkhu4pH4joeQugOicBAup38gDjbzWyCNNeKMpnXEkcMWv4doq92bVetwWn6FaL9pPAup9iXGBjpreGQmwres3Y+h5Z0EpU4gPyWhNtBBiNI6PhBfZydrzFg5BxeOomuGJcrc7FZlLhXs4GYnZ7uQ01kq9pcoKp2AiF2hpU5PLsl7mssVI4dK+rESG3Tv02LNHiy37dJDt1KEgWo1IKUH1PrNfFZ5/85CH0PBKEvCUej9OFZzD6Upe0zxXHhJA1FbyznOq32TQaKsFjsLZ65OED9nDeGt5BMjUoJaR0FFB5zYoLk4VhURNnBiraLabcbIqB2a7Rcw6QsJNpVAhWhWJSFWkUCtS74t6ZKxKbPg9ZtVfw9nVKyepGsrN5Sioysfp6vFObHdyjhkyUG/J4FThkcMDveRpI3WWSh24t0TC4thRNfbu0GD3Pj3+OKBF5DEVtIMNGNTXiIH9Degz2IBtG1f7Vf1dML7GTkV1CgHk5nxA5bv3fqQbLqnfaAYEbfV4HBI61F5Sw0kCyF9eqQzet10C9LdusptgslmECp+ulGxkarkKMaooRKmjoFOSOlEDtULT5gRYOHpULKjC/PfibnciPbhGoRYu5fS11t42Egbk7EAqvLIyoKIcMNHcLNWeYWqlTAzCDITGUAns36XD7t1a7NqvQ+lWPQwdbOjRrwpn9DGg32ATuvUwgbz0vLfVS9dgxY87hMpw1JjBOOeiCz2SkCebSuns9YhBo6SGcxsE6i6APDLyCROoIwFy8jA7zDCSAHObQoycOiKVEYjWxAgHD41SKwRYYzwT69iksCRjQdVKbFTedidyiY32Gu8UlicEQEuyL5AKj36V1Q4PFZVOoUQCixweNFqZ6H34Y0FpTp1UYu92Lfbu1eGXgzok7tfAMMCIc3tXoW9/I/qdaUCHDtWDR7wKod6RyegcXEtRJIiio2U4sHM9Row6z+n1RlPFUE+IZkhg9ZuLYEt6hlyNakEHoeZDH7CkQjTZzMIjkRw6aAozvUqHGGUUItSR1T0wdasQYOEQVH6+PVvQE9GKmuLP7hSpimg3didSk9HknTW9pZoBstRDInVZVKTvbArSLSa374N7Ndi3U4Mdf+qRs1MPGvOZfGYVBvc24Jnx5ejZzyRmPZDyuO9prJLJVONkQfXFxckQGQXoNE4nBUqfd1SO2Cj3nHzMBJqXALnK04/GfblvJMCKLaXIMxa4huWQCpFMBtHKaESqo8REzDT4mVSIbVlLw6o/9yejnseS3SmvKhd5xkKRu73YnainRLYlcg+nwbRVhhoXbMmJIFgvpbBAjj3bddi7R4t1B/SI3KVFUW8TRvY0ol//KvQfbETHNKtfGxH1tsQkptVqQwKv18kQEw1ERlbP98afYPV8mjl5ayFAjhsmuxkWmvvJbRb9KKUeUWQDU5EAU4NUiCq5uskFWDh6VCyoGvB0tje7k1j+wAQYjE6Hh7Lyujs8EF5Swx05RE4PWvyxX4eD+/TQ5SgQM8yA/n2qMHCAEb0HGhDh7ZpXfW/Izdts9BzAS70zEkx6HaAlbzpW3TXgSeYsbYmA5MBBAkwa4EzXR5odpwCLFAKMel/0C9fGgqoZbVTudieD1QhaG6mp7E71eaAaqz93d3ggZwdyepAcHqgdkscb2XwCbeXlMuzbocPuPTrs3qdFxXYdqjKtGNDHKZjI6aFLV3NAuxDNsWc2edqXYmJkiI1xetmRs0Mgh4tAbbr74QdwrKIAxoJyaBOdur/OkYl49fkXA2Vpt+GNfYbaOrjWxIfc5Z0OHGZYaVBf9SaXKxCp1CNaHS0EGQ1kJhf6UAxiDoegYgWJdOf87APZnSI0cX5St84gmg+PPPGEw0MFUFFRMyiI3K1pHFC0NrBUIsGWdUyFPTucTg+/H9Qh7rAajkEGDO1jwFVTS9Dv6VOIjasp15sU2ZeMpppeGgnD+Pga+xIt1dDYjYTU8R6nYVCboMusdBZ3qLGlcn4m0LIJkN1KI6M1w2gUeU1bJQF22pALS6VVzMhBg5mlQczkwBGligSNAWsJs3Cw6q/m3okjb7sTrY6qV7WNefbI4YGEEtmWysuB8jJfhweaEy5Yb4XGO/1JLuK7NPjjgB4F23WwRjnQeUAVBvcxoO8gI7r3NonxRF5oxak/+1KEXoboqNDblwpMxVhzeguWZu/A1rd/gOKsaI8mZR5KwXdvfIgl2b/gqR9fFQOhTAoFrEo1VEoNhqb1xAcjH/TIQyf7Sg7hrQM/IEKpFgsG0qKB9OsRnY5xHc/xSV9prUKhqUTM+K2jxQXlPPjTBxIHtAgC9P4THoh2ixjELC2aSgJMGgNGU0kFm4WDe1RhvJWV5goUmYpB8+xJ450S1LFNbogM5SVKMzxIDg80H54khCT3cF0ttp3cHCX27NBg7x4d1v+pR8xeDcr6G3F+LwPGjy1F//vzkJxSo1Lwbr8QjtX2JaqbftFRMiSnyJzeePVcLty7fH/n2VWnMW7R/dCVlUGZkokL03tBF9EJZpT7S45xHc/F2XMGwmgzocpqEFNZ0XGk0v8kwGK4gUqDcqsZ+cYKVFhMyLFbcbGp3K+g+uX0Njy2/DXIrTbYbTbYbDbxXMm698X28b6qx/W5W3HP5s9hE6sCKpGuVCNaoca5KT1xS68rfa7hZFUufi/YAyEEFWpE0HpOSi3iNbFI0XXwSR8oQFKPusezetSdRts/prFbNFEv/dw3aRBzaWW5mBuRPuBpk2bhIBuYXkVzIYZn/sN23aNy2Z0qT8FgM4lur/f6Tu43q6UfVxmA9avWo8eA81BeXuOqTfYkaS65YNdAtqFD+8hFXIsd+/XI2q2DwiBD0pkGDOxVhQEDTeg1wACN//e3KNrbvkQCMSra6fhAszg0xL7kr82kujhSmY1ukek+0TSwckvBLpyZ2F/MBkAJJt9xvVP1d5xUf05DstSj8ikgzAGS2oUmRPV2SaaqTxvysSFvJ6qsRhhtZlRZTSi3GtEjqiNmdBnv0zoSbP/Y/BnUFisMVgvMNiMcVjN6pPXGd6Of9Em/KGsVnl75DkxiFLNKeIfplGoU/5AFzZk2GLwYLXrjQ2ldPp+y2mNAa7JRhfv+uA9ittttwonjvLRzYbVYIQ/m9lvPhrU7GxXZncqMpcityoU0zx55xfh7YdSTZbMkpwlbaVmL3NOk0nPgdD6QbgD0EbS0fPAmlRTLsGc7zYunw5Z9Wsh26VDcw4yzextw7lnluOnmAqRlWFy9MH+lSW7ipNKjjexLCQlO+5JYAtxNL+4vf13D6ItuX/FBLM/Zis9P7Iby5GExMeDSWR+Iedfcy6E51c5OGuweBOoZ4BCczhTmGmcKj0RNdFJjN/BfIfWCpmVe5D/ST+h5yUOxZeJQPzH+g8Z3OgdnTu8teo8GixGVNvqZ8PTKN1GFUp9Mi46vxNP7V+KyDpkYlNgdgxJ6oWtEWqPWdPKphANaJQH6W9PRv+oliMh1XlIXhvKC2kWPqi3anWhwbXEJcDrP6YSwZd1arP55h3g2/K23RCq4o4fVordE0w/t3a+HLkuByCEGDOhjxMABBvQdbBCL7wV6wMhN3GLy9AKMjHDal/QRzoG1NMtDOLZzvr0N5ZUFiEnriWs7DcDYTsPRPTKzVatmw8GpMWVKvU73MqjX+dq/n8byk7/ht/zD2JN/Ctaik1CYLJh47iQ8NvB69+R83M4JkKAalDwo5OtRtekeFdmdCgyFEJ4t1fPstWa7E/VayquA3DxyhnCIsUORkTL8smwNPnuF1lt6R/yZfLz3HVhMaxCXeBn27NZg1349yrfpUJVmQ98+lejfx4DpVxejSzdzwCmMqCDJvmSpXp7a275EqrxQ9e6NNiM25e9CekQyukdl+vy5L5v0gpiR2ieCA8JOICOiE27seQVu7FlTVZGpxLWGU02o8+iN/V9jU+5hnJ3UDYMSumNgXC/Eqj0dWbzz8DkTCEagzQkqd7sT6fhpahKacr+1TvBIN49cyEtKgdxc56BXrZpsPjV6vRVLdwghRXOEifWWym7BR6/eC0WvWTi3jwFTJ5ag3xM5iE/wPy+e9ICQm7jFXDN+SdiXYpz2JZqGiNzEJWcMKU9D92WWCqzL3YafT+7AlhMHYc0/CXNcIp45d5ZfQUX3MBQb2xcCU/RQj7qNNat50mrykqNGoO3sDn1RYqnC/Jx9+GDXSqiLC2DTqvGvi27HhLQLAmVrNeH8DDX9rWoTgkqyO52uOo1Ss9OzK0odiQ5az7mzmh5v42ok54j8QqCo2GkAIjdundclFRXKUXrS1xBE7uLPv5EdsAHUO7OImcxrxjeRAExMlIkZIkJpX/LXiA8OLca8fRswNL0nHjnrKpyfMkS4v/pLy2FNQ0Aa/NzYF/GwxIGg32PVzSbV+5GKE4hXx/i9kEe3vQOT3YrBCd0wOKEnukd1Fh+YfhNzYLsk0GptVP7sTrT4oLdbZWu7q97OEdSr0el9J3M9dECNb+bH4cjyKMQNmo/cA0VweC217r7ekl/7UqQMsdGAVhd6+xK5iK869TuqbCa/LtWt7b5we8NHYOHxlViTsxu/FmTBVnASiiojzHHxWDjhSfSI6hy+irnkkBMIl42q1QmqClO5y+5kddiE+zG5lLf2mYNpIG5RMZCb73SO0Gl9FwykOfN+XROBTxbEwXpcg9FTijBxWomY9YHsVMt/cjpTjLt0MEZeNMrvMhcxMTWziYfKvkRPO6nyvvjrJ3yVvQvFJw9BazTD1qkL/t79bNzS+6qQ/0FwgW2XAD1L+4oPYVBCH78fnrdsegldopJwZnwPDIzviSRtQtuF0cqujAWVWFxPhvUnN7YJuxM9f6R+qzQAp3MBWtmW7D8RftzKKypkWL4wGgu/j4c5xoa/TyrCqEsqoHLT+FFZhioHtv++AQOHjhTLoNP4pehop2AKpX3J399OsbkMo1c/i5npAzAudSj6xvVskXbBxqq1/F17WwtryYxoDNp/9n6ODXl/IbsgB7Ki02JBMUVCGtZN+o9r3Fw470lL5hPO665L2eESVK3ORtVBG18XXi06DTlH0NIYp0/XOEfQbA3eW3aWCt/Ni8HO72ORMrISD/8zB/0GGT0cGsgzT1qQMCFehvRUGfr3kYGWSQ/VRjN1bCvYjRU527Ewex82jHvaZ9xZnDoaO8a/EKoquRwm4JcAaU7u6z8L91XHkuA6UZWDfSVH/Qop6p09uu19DEnshsHxPdArtqtz3ju/pXNgSyXQ6lR/Bwr3t1SWtbZLOEcUAUVFTgcGWkOJJgdw36hntHOzDl8vjEPhZj2GTCnG5OmlSE31nKaIZoCQ1oBKSZIhIecJiGwAACAASURBVJ6WUncvqfHHL+75FAsObYXt9HEYoqMxsFNPXJI+CFd2Husa4Nf4WrgEJhA+AuRG/+9dn2FFfhZMBdnQllfAGBOLMzP64LPzHw7L4NTwXU3LL5l7VC3/HvltIfV4yiqp9+QULMLl289Kt2YjsOKnKPxvcRy0ZjmmTyzC+CdPwdtxkWaCoBkoaHHCzAwZYqLCtxZTlEqH+86cgAuTh7ablYr93kQObLUEyI3+xWF3uNpP4/X2Fh9CvqnMr5A6VH4MXx9ZhcEJ3TE4vjc66pJc9u97HnkQR8vzXWXRAbn0v/LcC640HpF8EjICIf4GD1m7Wn1B5BwhzRxBwkqn8Rz7JF0grXS7aH4cNi6KRXQfIx6YW4Czzq/yGUhrNDjEUu9kw+rWVYboCP9jmuqiP6fViNec3oqlJ7Zjx8nDuGnQONzqx+HB3wSoUrtb674ufFrrtYWq3W2ZEXkFD0kcEBCV1W7FgeIcLDu8HZbCU8KOrExIxVV9z8XRinyf+SJpWi7ewk+ABVUIGUvOETRzRGmZQwgbGvvkz7vuwD4Nvp0fg2MrotHrsjI8+s4JdOtu9miN5CBBM0PQAN+MTCDKaxyVR4ZaTr4+ugxPbfwMkRUVUKRkYnR6b8y+8Cack3RGLTk5mgm0DwJ9YrrjiwsfFRdL9q9cYwF2Fh2ERqHCOqz2gXC6qgyDf7of/SPj0CUqARlRSciISEG/2K5I06f4pOeAhhFgG1XDuHnkkpwj8vIAo9kBjYrGJvk6R5B7+cbVenyyIB6WbDUunlSEy6eVIsZrUUEa8yQtYEgLCCYnOle29ajU68R9iQaL3QaVXCHUEtIgTkpOao1iUykGx/fjAZVe/PiUCdRGYMqdN+BY9xyPZAkH4jD73qtxrDJPqAV3VhRDW16O8T0G47EzbvBRL67L3YLf8w8gPTIJmfokpEWmIlmb2Gb+HtlG5fF4tIwTg9E5c0RhkXPsE/WeYvyshkvu5csWRGPR4nhY4q34++RijBpf4eNIQYKM1oyiLTVZhvg457RFtV0tuYZvyP0LtkEGz6ReagkxeNI5cbhnOj5jAkygQQQiFRrM7HpZnW1UVVYzNuUfwddHt8NcUQJNZSmsRjOmnDfZr2D7q+IEjBYDOkWkiJlbWvt40QZBBsCqv3qSk5wjcnOdQiWQcwQVe+K4Cgu/jsHuJbFIPa8CDz+Wg/6DjT410hx7BqMDVFZ6JxliY8TQEJ903gEkoC758VFYTx+H1VwOGbQeawl5p+dzoC3bX0J1f5mRf5LkOOE45HAuFeM2F6L/1P5Dx3c6F/Rz36gXQtPA+Vse46sjK/H1zuXQ0YBLAFV6HWIjE3DvsKmYmjHGvRhxTOrKtijMWFD53Gr/AbRarpg5Is+5IKH3xLBSLrIr7fhNh68XxKFoix5Dp5Tg2fnHkOzlXk7pTUangCI39S6dnQ4S/uxZUtne+1hVFO4783Jc3HEErtt3J47jtHcSPhcDqx0os1bAbLOgyFIGWqI+0OaAA/QfreIrl9GrQwaFTC5+9AKQQyGOW/Mkx4GuncODEyDvPtpCLcjVcpXfMWBU16MD54ofHZdaynGy4jSOV+WiR0yGaIv3/8ateQLFR/bAFBEDdWQs0iJi0DM6EbO7jUO/uB7eyVvNOduogtwqEjoVNDFsvtM5gpLS+kv+hIlwL/8xCh98Hw+VBbhmYjHGTSzzcS93d5CIipIhJRmI1Pn34KOBtutyt+Kzw2vw8MCr0CumW8DWBlpL6Ls3PgyYp61H0Oqj9Mdtc9iRqktCSkQK9KoI0DyRdoetek/H7uf0IWKHxWGFzW4VS1nQ3upwHlsd9PVrA3mH2UmiAaBZ6703KY7CFTIFCz1vQHweFgL0zOcY85FdkYPjVXk4UZEnvBVndx+L4YkDfep8YMubWFF4AmdGxqNzdAdkRnRARmSKcM2PUdXfTsA2Kh/E4Quw2mpWzSXnCBqz5G/mCGpBQb4c333jdC+P62vEgzfkY9hIX/dycpCorHTasuLjZEhKBHRa32sot1bixxPr8epfG2A/shfGqFiM634mIlQRvondQqQlGtyCnKvauge0k2OT3YQycyWUMiUyI9ORqE+EWuFcfp4QkOCgf43dSM3iKeTqL/QMNhZ6jb0PnL+GAK24S96G9Du7Jjjg0ZVdzkeP2MPIKs/Dn8U5WJi9H7LyUrx0wQ0YnXKWT74PDy1GpdWA9IgkZEYkC9tZB028UDfSOLO/yvLE34RPxkYGcI/KDSA5RxQWkYNE4JkjpOT792qw4Bune3mfy8owZXoJunbzdC+ntO4OEtIMEjTvXqDt4W1v4ZesA5jcYximdxmDzIhOgZL6DQ+1WsJvJS00sNJaBVqDTK/SITMiHbHaONAfrvu24ZcVGHlB3Zd5d88bzuO6CD36WrbCBqvNApqQ2WwzgVSVJpsJdthhqg6nsvz18qT2U2+P1Jj+enpymQI71v+Osy7wtKNIeXkfetVfa2L66r4v8X3OflSVl6KsohgRlRWwWa14e9q/8MKzbwqvSMO/jsJmtUHuT/XUwItt9zYqco4Qq+bmOufMU8iBKD8zRxBfEjrrV0bgo0VxcGSrMW5yEe788S9ExzoFm/s9IAcJmuKIHCTSOsoQF+vpIFFiLvO76unzQ24DhriXxMfBCNBLmdR7pCZN1MShW0w3RKujW51BWRIcoejpUS/P5rDBIVSaNb08m4N69M5zq90GW7U606nitIAEoclhQrGlHIXGYsFQr9SKufHaooE+2HPFcf4J3N33Gtzd1zPOYDNAIQuvKGm3PSpyjhAzR+Q6nSM0Khk0flRxdEvIvfynb6Lxww8JMCVYcOPkIlw4rtLHvZzSkoMEqQvJ2SIlFYiNci7XTi+PzXk78fnRtVh9eCvitHH4ddpbnnecz+pMgAQTCSjaOulTkaxPgk7ViNHQda657SckDzST1YAKSxWKDIUotpRBVv0tplVqoFfoWt2HQNu/a81/hdI4M+5RNfJe1Mc5gqrKOqbCwnkx2LMkFh0vqMBDT5xCvzN83cspbVWlAzSDBPXGMjJkiKh2kCB7yfTVz+PkkV2o0GoxqtsQvH/RvTgnaXAjr6Z9Zqe52sotVcJLqlt0FyRo46FSqNsnjDBdtUqugkqtQqQ6Wjig0EeWwVKFSksVSkzFKDKXwm63wS5zQCNXI0Kh81GxhqlpXGw7JRDe/loLgUrOEWUVwOkcZ28nmHMECbNtv+owb2EcCrbrcBa5l39zDMkpnrOX06VRWnKQoFV5JQcJvc7zojVyDUan9MK4IbOCeu155mr4WVu1UVHvidzLo9UR6B/fB9GamAatd9VSbVQNv+Ohz+nNiFzxI9SR4pcUkQRStxqtBhisBpQYS1BoLobFbBFekGqFSvS4VPK2+2ppq39jjX2SyKHLdtCOgzja2KJ88rfdpwkQk7iSc0RegVNvQavm0px5/jaTAfj5xyh8sjgecjswc2IxLn7mlFim3Ts92bWkNaCSEmUwaE5h/qmVmBA5An11vmMV7ux7tXcRfF4HAvQlX2qugA02sYprakSK+MqvQ1ZOEkYCZK8iNSv94nUJ6Eoqb6vRKbhMpSgyFQm1LKkLyZlFp9TwGlBhvB8tpWgaZybc0+cNCnmTQmqjWrHse9x938Ow2xyYO2s6HnjkCY8GlxQXYs6cOTh2/ATUei0+/d+76N3XOSFqz549odOT7lsOlVaNLZs2e+SlE/oDqW09Ksk5gubdo1VzyTlC72fVXKnw/Hw5Fs2Lw6bFsYjvZ8T0K4qFezmttuu9STNIyOBAgW4/lhWuxs9HtkNbXgJHl774aMgs9I/v7Z2Nz+tJgNSlFeYqkAdaemRHJOoSoVEGMCDWs2xO3jQELDYzqsjOZSbHjEKUW6uEnYv+htlBo2nuQXPUEq5xVCETVDabDf3698eKZT8ipWMahg0fjvlffY6evfu7eN33jzsQEx2Nx558FocP7sMtt92JFStWivhevXrht00bERef6ErvfRBMUJFzREkpkJvrtBWRM0Mg5wgq98AeDebNj8WJVVHoe1kZps4oRueuFu8qxbm3g8R7WZ/h660/YGD3Qbiu2yhckDKszUwq6RdAEwW63MuVWmREpCNOF8+2jyZiH+5q2EEj3IRbRvnhElQhU/1t37IRXTIzkJ5JigBgxrTJWLjgWzz0zxpBdeDgX3j04ftFfPeefXEi+yQK8nOR2CFZhJHuu75bZRWp9khIOfPSxLB6z6EzriJtVmD9qgj8b2EcZDlqXDq5CPcszUN0tP96yyvtsNtkoDK7p8vEEhvU07qn31V4sP/MFun51Nr05+7u5Qma2LC7l3vbX1wPBx+4CISDUVty0Ghtf2OuG9uKD0ImqLJPZCEtrWb9lYz0dGza7Km+GzigD+bN/wZnnzsKJNiOHT6MnJNZTkElk2H02IuhUClxyw3X4W831azK6c2XnBdKy+vmHEF5y8tl+OnbGCxZHA97Bwuun1qECy+qhMLP1ecb8rEweyV+zN0MmzUf2676AhFeXs/kIMFb4wiQe3m5pUIY4DvqU5CiT2b38sYhbVW52UGjVd2uZm+sn1d1w9pEtqXatkceeRx33XUbBg0ejIEDemHA0DOhUDi7P7+sWYmU1DTRwxp78Tj069cPI0aO9inyrjn3IzLGKRCjYqPQrW8fnHnWeSLd9s3rxV46//m7TdiwMRqnfp2ETqMqMG3610jvavJJP3j4SLz21//wy9pVsJLr7dn9cfeQC9HtlB57tmx0jdKnLynapFH7LfVcNLJ68syW1l6T3Yx+IwaJSV9P7zyKWHUMuowZIZpMX/K0STNHhOtcVAIgXOWHu/1tsXxS62/71fn3RddHepmVq5fAZDWh31lDhIPGqtVOM8HZ558jHDR2rt8qbmVL/3ts6+2jm7Bpw2bknMgRH54N0YxJf5OB9iGzUW3dvB6PPfEMli5bLup6/pknoJDLfRwq3BtCdqnt27YiItJz8sOnHn8YUZGR+McD/3RPLlRt837ZF9Q5grSHWzfqMW9hLAp36HD21BJMuaoUScm+7uVUuNXqnEHiP1nv4KKM3pja81xEabjH5AE+BCdllgqYbGZEqfTIiMxAjDa2Qe7lIWgKF9FKCbCDRsu/cS3eRjVoyAgcPXYcJ44fQUpqJ8xbsFg4U7ijLSsthk6nh0qtwf/efQPnjBgihJShqhI2mxWRUTGoqqzA8rUb8PSjD7pndR1HRvpxxyP3WAOwfEkUPlscL9LOmlSEcc+egqZ6XBN9ma0r2oDO+nT0iOwJs8k5poqmOMrMkOGTfrcKD0FXRa30oCXpzyX3cpp5PEmXgI4Rqc3uXh4O+0srfVQCNrulMqKB3TH008SgU1SaWMPJfQaNQnNJk8yg0ZL+xgLexDYWETLVn1KpxOsvv4Bxl14Ou92O2ddeJTz+3nvnNYHsxlvuwp/7d2P2DbeAHBL69OqBDz/8SMTl553GxClXiGO7zY6rr5qCMRdPqBPq/Dw5Fs6Pw2+LYpFwhgEP3JyPoedWiTpKzMX44cRafJz7K3R5B2GOSsIdXeagg7UHIiJk6NrFuQaUP1f0OlXOifwSoK8q6kHRxKhpkZ2QpOvA7uV+SXFgYwi0JQeNxnBoD3lDpvprClikx77vmbcx6pJR2L9Lg3nfxCJ7dRT6TSjF1BklyOxS417+Q85SfLbjv7DEd8Hk1LNxftRoxCs7iAG/SUkQHnxN0eb2VIe3e3mMLhb0MuGNCTQHAX8zaNA6YhSuVCjb/AwazcE8XKq/VieoYmLeQHlyMmIrJ+GyyUW4dFqpX/fySks5HA4Z7CbnOk7x8TIkJwJaNj+F9PmlP3pp9VxyL+8Y2alVzl4eUihcWIsl4D2DhsFmEupCnkEjNLeMBVX1zBSyXhfCoc1Cz+7psF6ejqPFh/Hdee+LRfIk1O5rQKUmyxAfBwRbA0rK1xb2TaU/l1bPJUGVqk9GckSyWD23pTNsqfaXlsStPTGqzUFDq/CdEaWp/sZa0jNR17aES1CFzEZV1wtpbDrtjOOiiCO/7cYwTT/cNOhe12qt0hRH5CCR3kmG2BjPNaAaWzfnB6TVc2nS0c5RGWJ6I569nJ+M1krAn4OGkWaKt9ISJ0UoMBX7OGi01mttze1udao/3RNdBO/E3cn47z+dzhg0xZHB6IBeJ0NqRyA6wrkGVGu+MS2t7e7u5em0eq4ujt3LW9pN4vaEnIC/JU5oUUqacJds5rRaMjkNkWZBOqdGSOcUR4ObKU78k8mgpHP6J8LkUFAJMme6kF9AExfIPSov4HKH2xpQUTKkZzinOPJKxqeNIEB/pCSgaBaJluJe3ojL4axMoN4E/M2gQcMtaKVkEkb0N+IA7WlFZekcznCHA5bqtPQ3RGlpNWUSdKQ6t8EOq8MCC6WrjqfSpI3EmSQIpTDau6eh8/YgDFud6k+6YWY7EB0jQ1IioPNVI0vJ2t0+FPpz+ipydy/voEuEVum10FYrJdue7C8NvUXMKDA56vlsXrfWNXtK4JQNiyFhR/9ImNGCdzWCUBKMJN4oSjq3w0YC0GF1CTtJGFI5JAC9hSGVIHp0tfQKqdfokDl7hwrRK5RDXj2WhwS4d8+QynSI1jXs2oPlanWCKnZnMjQqoGeHDshMC3ZpHFdfAgYbLT9ugEahRvforojXxbN7eX0hcnom0AgCkoqQBEG4tsYIQxKcJPxIUNKM+JIwpB6lhXqK1T3MULe91dmo9uTthzLA7OihhtMeyqOHltzLaXqjeHWMGKAbrYkVOvP2cP18jUyACYSWAM3farFYIJeHTti2uh4VC6nQPFSt1b08NFfPpTABJtCaCIRO5LWmq27DbSUbVbCN3MvJ5bbcWincy4cmn4kusV1bxRioYNdV1ziyv/AWnAAzYj7BCTR9bKvrUTU9orZRIwkmo9UkZi/vHdNDrJ4bTj1426DGV8EEmEBLINDqbFQHCve3BG6tog1k+CTvPVLzJWji0DEyFVGamFbRdm4kE2ACrZMA26ha531r8laTe3mppVy4oKbpOyI5IqnNuJc3OUyukAkwgWYnwDaqZr8FoWkAee+Re/mylctgtJnQI7obhicPRWZMJgspN8Rsf3GDEeCQGQUAUx3MfILzCUcs26jCQbWJyiTHiCqLCTTGnbYEdSy6x6ZjaNIQdi9vonvA1TABJhB+AmyjCj/jkNUgBJPVBLvdJkaMx6iixKSwkSo99KpInnsvZKS5ICbABBpKgG1UDSXXSvORranKaoC1ehJMEkgZEZ0QpY4UgonW0OGNCTABJtDWCbCNqgXdYZqahJwgCo3FYqwTzd/VMSIV/eP7YljKUAzscAY6RnUSnnuBhBTrz4PfUOYTnA/FMqPgjJhPcD7hiGUbVTio1rFMchuvsFWJubNoAkitUoNkbRJiNdFiAC6v81RHkJyMCTCBNk2AbVRNeHtpXFOFtQqk0qNNLVchUZuAOE0s9Co91ApNE7aGq2ICTIAJhJ4A26hCzzSsJZJgMtiMMNhMYqE1Utd10CQgThsnBJNGyeuThPUGcOFMgAm0CQJsowrhbZTGMkk2plJLBaJUkaApiwZ3OAPDk4eha1w3MX1RuIQU68+D31DmE5wPxTKj4IyYT3A+4YhlG1UjqRptRlRZja6lp+M1sUjTp0Gv1kOn1PN4pkby5exMgAkwAbZR1fMZ8B5kG6+KQbwuAZGqCOhUeh7LVE+enJwJMIG2RYBtVM1wP/0Nsu0cnS4EEw+ybYYbwlUyASbQ7giwjcrrlpNHXom5TIxjIlsTHBCDbAck9MXZycPRP7E/UiJSEamObpG9J9afe91Qr1Pm4wXEzykz8gPFLYj5uMFoosN2b6OiQbZVNoMYy0TMdUqtGGQbraaxTHqo5KomuhVcDRNgAkyACfgj0O5sVNIgW6vNKnhoFGokahN5kK2/p4PDmAATYAL1JMA2qnoCo+T+Btl20CbyINsGsOQsTIAJMIHmINDmbFQ0lqnSWuWyMdEKt/HqWPSN7YUhHQZhWPJQdInpglhtXJucCYL158H/jJhPcD4Uy4yCM2I+wfmEI7bV26hIMBntRlRZjGLpC4VMgXh1DDIjM4SNSavU8VimcDw5XCYTYAJMoIkItEoblb9BtnGaeESqI3iQbRM9OFwNE2ACTMAfgSaxUf24+BtccvkVkMtbplawwFQMGmSbHJ0ipifiQbb+HhUOYwJMgAm0HQI+0ujL+d+gT9++uP++O3Hoz70t7kppLFOfxL5IiUhBhJpXtfW+Qaw/9ybiec58PHn4O2NG/qjUhDGfGhZNdeQjqL74Yj62bN6Ebl27Yub1f8fZI8/F+/99HRXlpU3VpqD1BFowMGgmjmQCTIAJMIFWSyCgjaqwIA+fffwB3vzvh+jZoxuOHDuOW2+aizvvfqDZLlYmk4nJX5utAVwxE2ACTIAJBCUQDhuVT49qyXfzMXnS5Rg1eiwsFgt+27QePy1dhh3btuLNt/8XtIEcyQSYABNgAkwg1AR8BNW3ixbhgfvuxq5du3D/w48jsUOKqFOnj8C7b78a6vq5vBATYP15cKDMJzgfimVGwRkxn+B8whHrM47qqSefRnJKR1ddRkMV8vNOIz2zK0aNvdQVzgdMgAkwASbABJqCgI+NasjwYfhtwwao1BpRv9lkxIjzzsO237c0RXuC1sE2qqB4OJIJMAEm0OwEmsRGZbNYXEKKrlit0cJmtjT7xXMDmAATYAJMoH0S8LFRdUhMBDlUSBsdJyYmSqe8b+EEWH8e/AYxn+B8KJYZBWfEfILzCUesj43qvf++hemzZuOOfzws6kvumIKvPvkwHHVzmUyACTABJsAEaiXgY6OSctAAX5lMjojIKCmo2fdso2r2W8ANYAJMgAkEJRAOG5VPj4pasPSHBdizZy8MRqOrQY8/9ZzrmA+YABNgAkyACTQVAR8b1U03zsVX33yDt979WMwC8e2iJcjKPtFU7eF6GkmA9efBATKf4HwolhkFZ8R8gvMJR6yPoPpt83Z8+unXiImNwhP/eh6bft2I/YePh6NuLpMJMAEmwASYQK0EfASVWq8VmbQREcg5dQJKpRKFefm1FsQJWgaBkRdc1DIa0kJbwXxqvzHMKDgj5hOcTzhifQTVxHFjUVJciIfvuR1Dh41A1249cOXUCXWqe8Wy79Gvfz/06dMXLz73lE8eKpfmERw0aBCGn3M2Duz7w5WmtryuhHzABJgAE2AC7YuAw22z2WyO3zaucYUYDVWOkuJC13mwA6vV6ujVu7cj69hfDrPZ5Dhj0BmOP/fv9shy7z23O/71xCMi7NCfex1jx44Rx3XJSwkBMpvxFozA+rU/B4tu93HMp/ZHgBkFZ8R8gvORy+UOkiWh3Dx6VLSq70233e2S1BqtDjGx8a7zYAfbt2xEl8wMMSegSqXGjGmTsXDBtx5ZDhz8CxddNFaEde/ZFyeyT6Ig/zTqktejID5hAkyACTCBdkPAQ1DRVY+5cCQWffN5vdd9yj6RhbQ050zrVE5GejpOnjrlAXLggD6YN/8bEUbC6djhwziVnYW65PUoiE8CEmD9eUA0IoL5BOdDscwoOCPmE5xPOGJ9xlF99P7HeOW1t6FWzEWE1jkxLWQyFJaVB62fBgfXtj3yyOO4667bMGjwYAwc0AsDhp4JGhxWl7xS2XPnXIPMjAxxGhcTiyFDh7j+sCS3UelB4vMVghPzcDqY8PPAzwP9QfDfQ2j/HojpylWrcDwrS7xvnFYacRiy/wWcmaK+NWzdvB6PPfEMli5bLrI+/8wTUMjleOCRJwIW1atXL2zfthX79+6sU16emSIgSlcEvYylP0RXIB+4CDAfF4qAB8woIBoRwXyC82mSmSl+Xb/KbyvOOW+M33ApcNCQETh67DhOHD+ClNROmLdgMeZ/9bkULfZlpcXQ6fRidvb/vfsGzhkxREzRVJe8HgXxCRNgAkyACbQbAj49qgkTLgX1XGgzmczYunkzBg8bhlUr/Qswd1I//7QYd9//COx2O2ZfexUefvQpvPfOayLJjbfchS2/rcPsG24hTSL69OqBDz/8CNExcSLeX173sumYe1TeRPicCTABJtCyCISjR+UjqLwv+WT2cdx++x1Y9N333lFNfs6CqsmRc4VMgAkwgXoRCIegqtUDomOnDBw6fKReDeXEzUeA9Oe8BSbAfAKzkWKYkUTC/575+OcSzlAfr787brvRVR+p8Hbs/RODBvVzhfEBE2ACTIAJMIGmJOCj+vv0o3ddNiqlQomu3brirHMubMo2BayLVX8B0XAEE2ACTKBFEAiH6s9HUFVWlEOr04vxTXTVNpsNZpMROn1Es0NgQdXst4AbwASYABMISiAcgsrHRjV63DgYDVWuhhiqKkBhvLUOAqw/D36fmE9wPhTLjIIzYj7B+YQj1kdQmaqqPJafj4yKgaGiIhx1c5lMgAkwASbABGol4COo9FFR+GP7b66MO7dtAq1NxVvrIMCzUgS/T8wnOB+KZUbBGTGf4HzCEevj9ffWyy9g+jVzkJCcJOorzM3D1198FI66uUwmwASYABNgArUS8HGmoBwWswmHD+4TmWk5DpW6enLaWosLbwJ2pqidL+nP+YsvMCfmE5iNFMOMJBL+98zHPxcptEmcKd58/SVUVlagT//B4kfHb7/5stQG3jMBJsAEmAATaFICPj0qWiZ+586dHo0YNHgQdu7wDPNI0EQn3KNqItBcDRNgAkyggQSapEdFa4nQjBTSRuOo6McbE2ACTIAJMIHmIODj9Td29HmYMf0KrF21FGtW/oQZ06fhotHnN0fbuM4GECD9OW+BCTCfwGykGGYkkfC/Zz7+uYQz1Mfr78WXXsN7/30db77tnEppYP++yDl9Opxt4LKZABNgAkyACQQk4GOjopS7dvyOTz79FIt/WI70jDRMnXQp7rjr/oCFNFUE26iaijTXwwSYABNoGIFw2KhcsO+yFgAAIABJREFUPSpyR//880/x7cIliO+QiCsnXwayV61ZvaZhreVcTIAJMAEmwARCQMBlo+rZuz+27diN5UuXYN2ataIHpVQqQlAFF9GUBFh/Hpw28wnOh2KZUXBGzCc4n3DEugTV4gVfQa/TYNSYi3HzTXOFMwX1qHhjAkyACTABJtCcBHxsVLTMx/eLvsEX877Fr2t+wfSZ0zHtiqkYc/GE5mynqJttVM1+C7gBTIAJMIGgBMJho/IRVO4tKC4qwLfzv8TX3y7GqpWr3KOa5ZgFVbNg50qZABNgAnUmEA5B5VL9+WtFXHwi/n7znS1CSPlrH4f5EmD9uS8T9xDm407D/zEz8s9FCmU+Eomm2wcVVE3XDK6JCTABJsAEmIB/AkFVf/6zNF8oq/6ajz3XzASYABOoC4EmV/3VpVGchgkwASbABJhAOAmw6i+cdJuhbNafB4fOfILzoVhmFJwR8wnOJxyxLKjCQZXLZAJMgAkwgZARYBtVyFByQUyACTABJsA2Kn4GmAATYAJMoN0RYNVfG7vlrD8PfkOZT3A+FMuMgjNiPsH5hCOWBVU4qHKZTIAJMAEmEDICbKMKGUouiAkwASbABNhGxc8AE2ACTIAJtDsCrPprY7ec9efBbyjzCc6HYplRcEbMJzifcMSyoAoHVS6TCTABJsAEQkaAbVQhQ8kFMQEmwASYANuo+BlgAkyACTCBdkeAVX9t7Jaz/jz4DWU+wflQLDMKzoj5BOcTjlgWVOGgymUyASbABJhAyAiwjSpkKLkgJsAEmAATYBsVPwNMgAkwASbQ7giw6q+N3XLWnwe/ocwnOB+KZUbBGTGf4HzCEcuCKhxUuUwmwASYABMIGQG2UYUMJRfEBJgAE2ACbKPiZ4AJMAEmwATaHQFW/bWxW8768+A3lPkE50OxzCg4I+YTnE84YllQhYMql8kEmAATYAIhI8A2qpCh5IKYABNgAkyAbVT8DDABJsAEmEC7IxBS1d+KZd+jX/9+6NOnL1587ikfmIUFebhk/DgMGjwIAwcOxKcfvetK07NnT5wx6AwMGjwYw0ac5Qrng/oRYP15cF7MJzgfimVGwRkxn+B8whGrDFWhNpsNd9zzIFYs+xEpHdMwbPhwTJ56BXr27u+q4o3XX8YZZ/TF0heWg4RWz959cc2sG6BUKiGTybB29SrExSe60vMBE2ACTIAJMIGQ9ai2b9mILpkZSM/sCpVKjRnTJmPhgm89CHdMTUV5WbkIqygvQ3JSkhBSUiKHwyEd8r6BBEZecFEDc7aPbMyn9vvMjIIzYj7B+YQjNmSCKvtEFtLSUlxtzEhPx8lTp1zndPC3m+7Avj//QnpGJs4cOAivvvR8TbxMhtFjL8aQ4cPwv3ffqAnnIybABJgAE2jXBEKm+pPJapd5zz39OAb264U1q9fg2JGDGHfpRGw770JERsVg3dpVSE7phIL8XIy9eBz69euHESNH+9ycuXOuQWZGhgiPi4nFkKFDIH3hSLrj9ny+bes23HXvQ4IP81ghOLg/D8wHtf69EDRixs+P7/MjHijm4/F+oZOVq1bheFaWCA+LZswRom3Lb+sc48dd7Crtuacfd7zw7JOuczq49JLxjt82rnGFjRkzxrFj66+uc+ngyccecvznhWekU9cegOuYD/wTWL/2Z/8RHCoIMJ/aHwRmFJwR8wnORy6XO2w2W/BE9YytvRskfULUsh80ZASOHjuOE8ePwGI2Yd6CxcKZwj1bzx5dsGzZchGUn5eDw4ePILNLDxiqKlFRXirCqyorsHztBuEB6J6Xj+tGQOo91C11+0vFfGq/58woOCPmE5xPOGJDpvojz73XX34B4y69HHa7HbOvvUp4/L33zmui3TfechcefexJzJk7F4MGDaKuEf79zGPCy+/40cOYNHWaSGe32XH1VVMw5uIJ4bheLpMJMAEmwARaGQGemaKV3bDamkt2Bf7iC0yJ+QRmI8UwI4mE/z3z8c9FCuWZKSQSvGcCTIAJMIF2Q4B7VO3mVvOFMgEmwATCT4B7VOFnzDUwASbABJhACyMQMq+/FnZd7bY5pD/nLTAB5hOYjRTDjCQS/vfMxz+XcIaGzOsvnI3kspkAE/AlIJfLhfesbwyHMIGmIUBzvNJzGO6NbVThJszlM4EwEaAXBL0oaEJn3phAUxMIZIsKFN6Y9oVfFDamdZyXCTABJsAE2j0BFlRt7BFg/XnwG8p8gvPhWCbQEgmwoGqJd4XbxASYABNgAi4CbKNyoeADJtC6CLCNqnXdr7bW2kC2qEDhjbl+7lE1hh7nZQJMIGwEbr35Bjzz1KO0aEKtdUhpw7LERK21c4JwE+AeVbgJN3H5ZIPhuf4CQ29LfFpyj6pXr154953XcOHo8YFvBse0agKBek6Bwhtzsdyjagw9zssEWiCBhfN+wJjx94gfHTdka2wZ5DLvsAfuCVmt1oY0q93lYU7Vt7ye61c1a3JeOLFZ8XPlLYyATCZz2O12j1Yt+Pp7R3LK2454uVX86Hjh/MUeaWo7aWwZs2bNcCjkckeMTueIj4xwvPTvpx1Zx/5y0IJ6H7z3pqNbt+6OC0ddKNp+5bQpjk6d0h1x0VGO80dd4Ni/9w9X8+bOucbx6CP3iXS0WGHnzl0dL7/4rCMlpaMjLT3D8fEH/601bWqqlPYdV9rCgjzH5ZdfJuocevZwxyMP3+sYecH5PiylDMHaaKiqdPzj7tscPXr0EOVRORRG26YNqx0jzhvpiI2JdnTt2s3x6YfvOhx2u2PU6FGCg3TvPv3oXcd5F9bUT5zefP0lR58+fR29evUS7brrjptFGcTpzGFDHb+uXyU1TyxSSAvV9u7T2xEfFSnis08cc9x2y99E26R6KANd96svPR/wWl2F1uEg0AKJgcLrUGTAJNyjanffaHzBbZnAWx+thiXvRsghEz86fvODNfW65Lc/9l+GI3AHyaP8Tz/9Ct179MB3SxaisLwC9z5Idibntn7jRuzetRPLly4VA5UnXDoOB//ch9yCAgzu3wczrp3tmm2DemXug5lPnzyB0vIyZGcfxwf/fR3/uPNelJYUiYL9pS0pK0VWlpT2Plfa2267BZGROpw6fRpffPQB5s1f7FGP1FZpP+HS8a42njmgr0cb77v3LuzcvR/rflmNguISvPLCMyDVV3bWUUy4dBLuufkGFBQWYeuW33DmkCFA9TW5X5dUj/v+hyU/49eN67Drj52ibWcPH45tWzeLOmZdNRVXXT0XZpNRZHnl/57DvAXfYcnihSgsK8fnH74HvT4CN1w/FwsW/eTiWViQh42r1+KaWXOCXq97O1rMcUAR1gIjuEdV+03hZbKDM2pLfPz1qEaPu1v0pBLlNgf9RM9Kf7c4lsJq28frfcugcr06b0FBU09gzaqlrjRSj+r40cOuMO+DkuJC0esqKy0WUdfPvdajRxWr1zusVqsrW2pqJ8fWzevrlZby69Vqx+GD+1zlPPbP+4P2qFwJHQ6HextpuXVq095d29yTiOPnn3nCMXXyRL89l9FjRjs+fP8tV5y/HtW6Nct9ynQPiI+Pd9Xbt18/x5Lv5rtHu44HDBjgWPXzEnFOvbTLLrvEVa8rUQMPAvWcAoU3sBqRjXtULeaTgRvCBBpP4La5o6FKeg92OMSPjv/38Sjk2+R1/r3/oW8Zt98wijoDjd46pmW6yqCVwB984G707tMH8THR6Ny5i/j6LyoscKVxP4hLSfWYVy46OhLlFeXuSVzHgdIWFuTCaLWik1s70jp1cuXzPqA2PvTgPX7bWFSYhzKjEZ279vTOhuyTJ9GlS4ZPeF0D0tJrOFGel198FgMHDhSc4mJjUFpSgoKCfFHc6WPH0K1HL79FXzvjCnz2xReC62fzFmL2NVf5TdfSA3lS2pZ+h+rZPvb4Cw6srfOZOv1yyOTf480P7hMgSMBMuXJicChesaEoI5BUc1d5ffXZh/hp6SqsWPYj0jO7CtVcfEIHl6qKmuWe3quZPqd1SZuQmAytUomT2cfRrUcfUUZWdrZPWVJAsDbGJyQhWqvF0b/+RL+BQ6QsYp+elobfft/mESad6HVaVFRWSqc4dSoH3mpVmaymD7Fpw2q89Oo7WLViCfr0GyTyJSQkwO6wi+PULp1x+OAB9O470FWmdDBr9lz07z8Qe3dtw6E9ezBx6vR6MZXKae59DY3mbgnXzwSYQEgIkGBatewV8auvkJIa0Ngy4hITcPjQIak4v/uyinIolErExsWjqrICDz90v4eQojFRdR0XVde0ZD+aOOVyPP7kEzBUVeLQn3uD2qjKKyuEzUlq4yMPPwBJqtDwgJlzrsXd9z6I0znZYoLgLb+tg8Vswszr5mD9mrVYOO8zkOdeUWG+EBYE4oyBfbHo+6UwGqqEkPvk8/l++UiBZaWlUKvVSEhMEmU//eQ/UVFWJkXj+llX47GnnsORwwcEr327t6O4yNkr7dgpA8OGD8fM2X/DlCsmQa3RuvK1pgMWVK3pbtWhrTyXXXBIzCc4n1DFPnrfXfj3/70OUlORsZ82uZfucPacv6NzZjo6p6Vj8JlDcO6IEVC4LRlBPST3XpJ3fve21iftm2++jbKySnRKTcW1c2/A1CmXQqP0r1y6bvbfkOnWxhFnneWhfnzpP6+hf9+eOHvE+egQH4cHHnlCCCxSLS75aTH+7813kZQQj2HDR2DH9u1CyP3jHw9Ao1IjLSUF191wI6ZPmxj0Oi+6ZBIuGjsSvXv2RtfuPaHVaNCle3fX5d9938O4cupEXHr5ZCTGRGPOTbfCaDS44ufMnIHdu3djznWzPOpxJWgFBzzgtxXcpPo0sS0NaK3Pddc1bVvi05IH/Nb1frSEdGQny8/Pxwcfft5qX+TBOJLq8Lq5N+PgwT9Den2BBvYGCg/WxtriuEdVG6FWFt/WbTCNvR3Mp7EEW3/+wwf3gdRjpC7cvmUjPv5kHqZOmhTSl3hLoWSxmPGfl1/H7Jmt04lC4siCSiLBeybABNoFgYryUkybPgsJkZG4etYNuPeum3HZpCvb3LUfPLAHKfEJyCkswt3/eKBVC2JW/bWxx7MtqbbCcWvaEh9W/YXjCeEy60ogkIovUHhdy/WXjntU/qhwGBNgAkyACbQYAtyjajG3ghvCBOpHgHtU9ePFqUNLIFDPKVB4Y2rnHlVj6HFeJsAEmAATCDsBFlRhR9y0FZANhrfABJhPYDYcwwRaKgEWVC31znC7mAATYAJMQBBgQdXGHgQeJxT8hjKf4HyaO5Z6vF27dXdNnXTGGWdg47qVfpvlndZvoiCBvHx9EDgtLMr/vCEtrJHcHCbABNongT/++CMkF/7pR+/i/U++xLo1a13jid7+7wchKZsLCT8B7lGFn3GT1sA2mOC4mU9wPhzb+gnYbLbWfxFeV8CCygsInzKB1k7gmjuvwYg5Yzx+FFafrTFlvPjcU7hy2hSX+o7qvevOW3DXHTeLSVk//uAdDBgwAAnRUejduzfe/+/rAZvWq1cvrFn5k4in2cavn3st4uPjMWDgAPy26TePOv797FNi3Sgql+K/X/i1yEczNNx52z3YvGEjEqOjRH6aPonKeuyfNTO2/+/dN9C3bz8kJCZi4sQJYkZ0qWHkcv3u26+iX79+YqLd22/9u0fdUjra07RMI84bKdKlZ2TijttuhNlsciU5sO8PjBt3sainU6c0PPf046IsEjDPP/ME+vTtI9gMGT4Mp05m4cTxI2IGd3cBNHrMaHzw3lsi32cfv4dzLzgf99x1K5KSkvHk4w/h2JFDGHvRWHTokISkpBTMnDndtcIxNYSWOZk6ZRJSUlJFGqmNiYmJ2Ldnh6utBfmnkRAVhYL8XFdYcxywoGoO6mGsk20wweG2Bz5Hy3LxR+ejHj8Kq892rCzPIz+VR2U4UPt69NfOmo3Vy1agsnpRQ3rBLlq8FNfNmimWYk9NScGPP3wnlk3/8P238NB9j2DXjt/9Nk/Mii53rtj45JP/xF/HT+LIX4ewfOmP+Oyrbz1mZO/VsxvWrV0lyn3ykfsxd/bfkZd7Cj1798cbb7+Ks0aei4KychQVFQn1n/uM67+sWYZ/Pv4c5n35MU6fOokumWmYfu1MD2G0ZOkKbPp1A/bs2YUFi37EyuU/+G2zUqnCWy+/gMKiYmzasBa/rP8Nb7/5qiiLpm+6aNxEjB97IXJOZuPgn/sxfvw40Z5XX3reZ0l5nU7vtw73tlOC7Zt+Q7duXZGTcwqPPvYvWgwdjz50L06dysb+fbuQnZOHJx7/p0sgTrh8MjpnpuHokcM4mZ2FWTOvhVqtwYyrpuCTTz9xXfeXX3yKc0ddIJYY8duQJgpkQdVEoLkaJtCcBH47+Sv0T3YVvxvXP+q3KRQupdl0cqPfNHUJ7JTeGf2GDcXihc51ln5ZvQwReh3OHHauyD7usinI6NxNHJ9z3hicP3Y01v6yttaiF3+/DI89eC9i4xJA6yzdcevfYHdbcXDKlTORlNxRlHPFjOuQ0aMHft+0QZzXtq7V5198iTmzp6P/GUOhUmvw/L9fwtaNm3DyxDFXux59+H5Ex8SJ1YEvOJ+W7djheqG7EgEYOHi4uFYakJ2W0QV/m3Mtflnn5PnT998hKTUJtDQHrQ0VERnl4vLR51/j2ScfdS3o2Kf/YMTFJ7oXHfA4LbMzbrvjXtHz0mh1YtXhC8dcApXKuY7VvXfcgnW/Oj8GqMd3+nQu/u8/r0OnjxDtGD7iAlH23DmzsXDRT9KSW/j0q28xZ+b0gPU2VQQLqqYi3UT1sA0mOOj2yufsTueg6skj4vfeec/4hUThUpoRnZxCxW/COgRePW0Svpy3QLzIv/jqK1zltubSzz8txlnnniNUX7Re1Yqly1FYWFhrqTnHjyM9o7MrXefOnV2OERT4+cfvY/DgwULlRuXu370b+QX+l7V3FVJ9kJNTgM6ZNcu/6yMiEZ+UIlRvUtrklJol6yMidKBFFf1tfx3aj8svvwxpaRli6fgnH/sXiqoXOsw6kYWuGWn+siHnaOAl5f1mcAtM7pTqdgbk5+Xg6hnT0LlzV9GGObOuR1mJc7HFE8ePIy0z3WNdLSnzkOEjodGqsXHdCrGo5JGDB3HZxGkenKW0TblnQdWUtLkuJtBOCFx51TX49Zd1yDl1AosWLcHMmdeJKzebjJh+5TV4+J7bkZebi+KSUlx0yTi/PRNvVCmZmcg6ftQVfOzYMVe+7KyjuPWWO/Hft15BUXGJKLfPgAGutKQqC7alpnbA0WM1vSdacbgo77TouQXL5y/u5lvvRO9e3fDngb0oKi3Dk08/DrvdqTLNzMjAkaxsf9kgLSnvHUm9HtpoRWJpy8vNE+o96dx7UclHHnkQcoUcO3dsEW34+LMPXb3P9MxMnMw6KRZ4lPK772fOmIZPPvscH338IaZNm9wiVgVmQeV+h9rAcXuwwTTmNrUHPl2ik3HGsS4ePwqrz9Y5OskjP5VHZcgQ/IUv1ZHYIRnnXngB5sy9AZ27d0f3nn1FFDkVVJnN6JCUJL7oqXe1buVqKVvQ/eSJ4/Hci6+gpLhQ9HTedHMvr6ysEPYqWq7dbreDHDYO7NnjEmQpKSnIP50LWp9J2tyXr7/26hn47PNvxHLxJEwfefh+DDlnBEiN6W8Lpkosq6pAdGSUUKvRUvfvfviZq4hLLp+M/Nx8vPbKCzAZDSCbFaniaAu0pDyxzOzcBZ9+8j+xrD1dG/V0gm3l5ZWI0OsRFR0rPhb+/cqbLhakgk1O6YCHHrgHJJCpHb9v+sVV3KzZc7Houx/x7YIfMHf2dc3em6KGsaBy3R4+YAJtg8CXr3+JTR+v8vhRWH22UJRxzfQrsGrVasy8corrZRcZFYNXXnkBV109B/Fxsfj8668wbuIlHk3z7h1IkU8++Sw6Z6SiS5euGH/JBMyaMc3lTNGrzwDcfs9tOOuc85Ga2gm7du8RnndST+rCMePRo0c3pHbsJDzdSNC4OySQPeepxx/ElVdfh9ROafjraBbmffGZq93ebXLPK7VP2r/6wrOYv+B7dIiJxg0334ZpUy5zlUPX//PSxfhp2SrRlt59BmDlipVCiARbUv7dt17G62+9jw6JCeLa/r+9a4GOokjXf0gIEALhlfDIE+SdkAx6lPWuux7v9R5XWFaRCLi6vgBFhKDeVe+eVQR0vb7WFRARZWH3KB4X5CGKgvJwQXmIEOSREJJAXoRAEkJIIG/6nq+G6vT0dE/XTCZkMlN1TjJdVX9V/fX13/9f9Vd1NTaG0NVBgxEv8+fNo4OHMyiqRwSNHX8XTRx3BwV3sKt77GD8cuMGyj2ZT/EJgyg2biBhjY4b3+iYeLrhhuspqEMH+sUvb+PdatNfeXp6m8Lv/caxBhMIswZPkfMnfOTp6Z5KgSxnhcD0aQ8SdmfOf/l11cjqy5idkm6Wri/vTlyeTOEOWpJWIiARkAj4OQJ4b2vt2o3086GDpkbqWkMgXX/XGvFWbk/OplwDLPFxjY/MDWwE5r7wHI0elUJ/fCaNba33FTSk689X7oTkQyLgJgLS9ecmYJLcqwiYufjM0lvSuJxRtQQ9HywbqO8Jid4KiY8oUpJOIuA7CEhD5Tv3QnIiEZAISAQkAgYISNefASgySSLQHhCQrr/2cJf8l0czF59ZekuQkLv+WoKeLCsRaGMEYKxkkAj4OwJSyv3sDss1GNc31J/wwQkM/HQFb/7u+u6bVqnXmzy2ZV0SH0WVj2s1UPKqofp280ZKTEqkESNGEr5Jow/lZefozt/cQbbRNkpOTiZ8dZMHq7KcTv66RuDATwdcEwR4rsTHWgAkRq4xOnjgoHqKg2tKmestBLxmqPDNmdlPP0+bN31Bhw8fok/WrCd8sEwbFi96m1JSRtKh9EO0Y/tWevrZP7Ozq0TKauuR1+YIVFReMM+UOSTxsRYCiZFrjCQ+rvFpjVyvGSocrDgwPo5i4wexb6BMSb2b1q39zIHnAf37U9XFKpZWXXWR+kZFUUhICDuU0aqsQ0VejnjiDhIp44rGLE+fro+j69o07bWXYVGr86QNkTKuaMzyjNL1aVZxtWNevNC3KVK1SBlXNEZ5Iml6Gn1chHd3aTxpQ6SMKxqzPKN0fZpV3N3+i9Dr27QqI0rvis4oTyRNT6OPW/Hubr7XDFVRYQHFxPRT24+LjaXTxcVqHBfTHp9NGVm5hM8zX59sI3zREkGkrENFXo5s3bbN7RpFyriiMcvTp+vjYFSbpr1GXn5Bgdt9sSqgb8OKHvkiZVzRmOUZpevTXMVbAx/R/upx0/Opz7eq16i8SJqeRh9vDYz0bRj1VZ+2bft29ul6fbo27ooGeVjL0gejdH2aq3hr4AMe9W3q+dbHReld0RnliaTpafRxPa8tjiteCuvXrFKmTX1QrW3VP5crs2ZOV+O4eHnen5W0WY+ztFO5WcrQYcOUqosXlHWrP7Isi0JXP8Aif/GdafknMZAyIGXAR2WgqanJQfe3NOK17ekxsbFUVFSiGk6MOmKjm7+IiYw9+w7Q3BeeZzQJg4ZSbEwM5ZzIYDMsq7IoZDQ6UhuUFxIBiYBEQCLglwh4zfVnu+FmOpWXTzh5t6G+jv619nO6+56JDqANHTKQNm/ewtLwqeScnJMUP3AIiZR1qEhGJAISAYmARCBgEPDajAqbIha9/TrdMXY8+8LmQ/dPoqHDk+iDpQsZmI89MYdeeHEePfzII2Sz2djs6LVXXqSevfqwfKOyAXMXZEclAhIBiYBEwBSBdnWEkmkvZIZEQCIgEZAI+C0CXnP9+S1CsmMSAYmAREAi0KYISEPVpvDLxiUCEgGJgETACgG/MFTZWcfoiccfpUn3TqAVHy6x6nPA5X+5YTU9Nv1hmjJ5Im3/dlPA9V+kwwV5uTRt6oN0b+oEEfKAobl8qZoefug+Jj+ffrwyYPrtTkchO9OnPcRkR+5MdkZu0+dr6PHHHmH6Z9s3X1q+G+dcg33Ld0u3uPtMeezdvzf1bp/hx9cYuVBRrkx99A++xpZP8ZM6UcqP9oZ8/I8Pla+/WMuSJk+aqCjKFW22vNYgcG/qBOXKFYmPBhKHS+gfvGvrCUY+NaOa+ugD1L9/NKWkpDgYVZEDa7/a+Bn9dtxYun/KJIey/hRpCT7AYcH8ufRUWpo/QeLUl5Zi5FShHyZoMdLOAIyes8KiIoqLT2AoBIdAXQT5ISLOXZo29Q+qLtJjlDQqST14W5vnXIv/pgCfAVd1tRaDrVu+II7P63+Z5/Du68sLXqK0WbMoKMgDGXIweW0c2b1rm3I4/UclOTlZ5aSxsVEZNny4UpCXq9TX1ykpthQlK/OIgpHenNkzlDPFhSotLsaPH+cQ96eIp/hgBPPsH9OUHVu/8ic4DPviKUa8skCYUWkx4qNbPGfDR4xQ8k/lsOfMNtrGnjOcMMNnVFMmpwbMjIpjlJKSos4AjDA6nnGYi44SSDMqM3xGjBzpIEPABzL23LNzlO3fbvJYfnxqRnXzLf9JPXr2dhiGmB12e/9D0+idRUupX/8Y+mHnVpozewbzg/7Xrbc4lPeniKf4LF74Jm3792761+o16ntt/oSLti+eYlRxvoxmPP4IHT2WZfiJGm0b7f3aDKOEuFj1UOnJE+9ih0pPSJ1Ca9ato5kzptKE8eMCZkZlhpH+8Oz169bS+fJSemLGo3Tk6HF64/8WOMwi2rusmPFvho9WhnAwOfB5d9FbTP+sXvMZfbB0kUf4eO2FX7MOtTTd6MDaPfv2OVT7y1/fTvgLxCCCT9pTzxH+Ai3gw4L4sJsVRqDDi+fvLwvczQIcI+6WwaHSeM66hHWlv6/4ONBEx7C/wCg6ui8pyhUKCgpmR8ABI8jO0vdXGJYJpMTTRYVOB5Pv3ruXFr27jGbPebZFUPjUjMqoJ0FBPs+iEdvXLE3iYw51UVEe4VtnVhhxOvOa/D/HCiP/R8C6hxyjoqJ8JlfWJQKGIHrMAAAQCElEQVSLguPTGr32eSsgcthtawDTXur0FJ8rV5ro9On89tJNt/mMiOhOISEdKTg4mFxhVFdXq9KhkaamRrp4Ufzjk6AvLS2hkpIit3n0pQIcI74wbnSotBG/ly9fImAYCAEYVVZWs2/oQa4KCgudDt42w8Hfnzf0OzomxvJgcjN8rNJ93lDJA2td30JP8IFyycvLoerqi64rb8e5SUkjKCrK/n00VxidO3dGpauurqJTp04Q0kQCDFR5+TkqKztLTU1XRIr4LA0wyssvUA+VXr1uo9Oh0lrmMVOFcc7OPhYwhgoY9egZQfW1terB23dNuIfBAlkoLDxJxcUFdPJkFh06tI/9IrOtnrfKyvOUkXGIjhz5iXDNQ21tDbtvJ04cY7zx9Jb+6mUIB5MDH+5OblH96pYVH7iYMnmiEhMbp3QNDVUSBg5SVi5/j3G1ZdMGBbtJsPvv1Zfn+gCnbcOCN/GprKxQjh9v3rEk0qOqqkoRMkWUTqgyN4mAUWJSknLXb39jKUO1tTVKUVGeQwvnz5cq2dkZDmlWkaKifAV/7SV4U44yM39WIEvtMbiS0/umpDroohUfLlEgL1u+3qCMTExUdRF2tJ08ecLp/gOTI0cOqLB48ryphVtwce7cGeXYsXSnGoqLC1l/nDIEE4zwARbffPW5is9fFrzIdvy5wlmwOUUeStsiM99+C8O9deZMIQ0bNkqoE9hwgBFjfPxgl/SidC4raWEm1pz69h1AHTuGuqzJiK6ioozKy0tp8OARLstqM0+ftn9VOTo6TpscENdZWUeof/9Y6t69R7vqrydyaiQvmFGfOVNEiYmjnWYOoI+K6k+hoZ2YO9md581bYNbUXKKsrKOMP+3zUFBwkuLiBnmrGdN6PMHZqDKfd/0ZMS3Tri0CDQ0NzEhB6FwFUTpXdbQ0r76+jlWhfSiN6hSlMyor09o3Ap7IqZm8wE3cs2dvJyMFhDBYwq7TtgxdunRl67RaNz/ctlhja+3gCc5mPPn89nQzxmW6dxCAv7qk5DTbcltfX099+vSl3r0j1covXCgnpF+5orC1CL5+g40Kva5+SwzEInRVVZVsAwd81jExA6msrIQ94BBo1NWzp/3bZGrjRNTY2MBGhGFh4TRw4BBtluE11gowirUKVnTYIGBfe2pi/QZ/vXtHWVVrmA8ld/ZsMZvh4aX8uro6psQ6dersQI82gS/S0W8swAPnsrJzlJAwmMLDuzvQW0VQHvcWyhJ/uNfoQ9eu3RyKgj/MDIKDQ9hmEswqcc+Tk290ULQNDfWE2SP60KFDMONTW1FNzWXCLALtJiQMYX1GPupHu1Do7gR360O7mN1gQAUZa2xspH79oqlz5y5qsyJyqhJrLozkBfcLf2b3xWiwZPW8oUmrfriLC/jD+it/vmC0unWL0PTOfgl5x7PIjRh4hbHVy6mIvHiKsxNTVxOkoTJDJgDSoXigGGNjBzLhRDwz82eKiOjBFCQg6NHDrlwgxESKqREQocPDMWBAHBUU5DJXCHcjYoR34sRRplx4PRx+KB0oHPBmFUADerhaXAUrOhgJ9Be82hVeAx0/foTVa/SAu2oLdeXmHqdBg4apDzwedKQNHjySOnbsyIqDbyzCw+XIFRwW5rELDzhpla2r9ngeyuXkHKfIyL6qgoIbKDs7k7mBuDICfX5+Dg0aNFxVUFBOoANOXEnhHmVnZzBZ4RhAYcIQ8tClSxhTbLi/paVnKCYmgRk6YAC56t49ghlDTm/160596G9ubhYbaHGDCP5ycjKYe5tjyuXLSp61vJnJCxQ5Aq9bW8boGvVYPW8i/XAHF/ABQ2Xvr50rGK3+/WMcWMR9vHSpiq67briajoFlfn4uDR2aqKbhQkRePMHZoRFdpG3npTpmZPTaIoBRMTdSaBkPHP5qa1t3uzHa1T4oUJqYyZ0967zbDkYnKel6GjJkpCU4UAIisykrOhiSfv1imJFCo5jV9OjRi806LJnQEWCEj/UbrvCRjT51796TrRFycigJ4KBVejAIlZUV1K2bfas9pxX5xQwJL6byUTTK4D0XGEaM2HmAYoRC1+7Mwv2BkdEGKDrgwI0U8qAwO3VyHhSgfHS03UiBDuXscmVX7Np6Ra5F6sMIHsaeGynOH2biWiUt0p6exkxeOI74urlIQD+snjd3+iGCC/iCocLMDwMGBMiF3iUJmYScaQNktrb2sjaJDZxE5MWhkBci0lB5AcT2WgUUo1ZBoR8QYP4Atla/tKN53gZmDBjxGwUjPvV0eAjBt9Yg6GkQF6GDctPzaOfP8aE1ql+fBsVj5BoKDw932DKMcnCv6gOUkScBBi483NG9gz6MGJHiYAxx/yMierEZLRQ63gtD6No13AFLjK7Dwx1dhmZ8Gd0vtINZmSdBpD5sDgoL6+pUPeTh0qVqp3TRBFfywl9whbEXCUb90D9v7vTDqD4jnDGggFsXLj8Mwow8DnBtR0UNYLPo8+dL2asHcHfq+yYqLyJ4uEMjNhRwp0ZJKxHwAAE8SAgYFetHeyLVYdQbGWm9NiVKp28TBgO8uRNAD+WsnSXx8khDHu8vZiqY7cAg8FkLFIbIDJHXqf2FguVuRW260XV8/HXMaJaWnmVriBERPdlOPq3RhwHDzMhXA/pLZH/5WssjFK27blNteVfywu+r1kWqLevJdWv1A4MMuPzwvh+XLy1/aBc7AXGP4S7u1SuSGTWj2aiIvGjr9sa1NFTeQFHW0WIEoAgxQvXESGENC39WCkmUzqgz4A8jWHcC+mI3cM4zCfBiz7c7NaBQYRguX65WX8KEsjBSKiI88I0RIrSgwawKf3ARYfMGXgaFu5Vjir639kxblFcjOvQXhiMy0v6StxGNu2lW8oKZCmYYeKHXaNaMGQxmL0YzGDNeWqMfaAv8wR0MfrWbpTgfeNE9LKwbib5iYSUvvF5v/UrXn7eQ9PN6oKj0bgDtllfefRE6I08JFLSZawkzD33bvD38YuFeREFZ0YUEB9O06dNp5cqV7GvRtTXNrj4snOt3y2l5MLvGxhQoMn1AfzFz4QFx4In1LPQFGxpgpPBlgP17d3IyhgPWs1zhAWI+guYFc7MzadzYOykxMZF+/8AUmjzpHiorLWGbWjCL4wHGEsoKygzuQx6gxPjmAZ6GXyhzTwL4F+mHaN1QxGauYyP8ReTUSl4w0MDaJdy7RgH4uWOkUIe7/TBq1ygN9eL+GckNdvrBPYq+WAW4JkXkhdcjgjOndfUrDZUrdGSeikDnzmEOgn7hwnnDHVwidFigrapqPr6Jj+KxgUEf4FM/evQg23Gmz0McRgxbqTG6dRVE6CLCutDyDz+kSZNS2VE57y9ZyKqEa6eq6oKpG87VTAMvw54/X+awaxHKHfghjwesi4WGdqYTJzLo8OH9dOxYOuXmZlL6oXT6ftf3nIwd0YPdd3h51Cyg/j59+lF9fS0zNnW1NfS7u1Np9szptHXrZlqxfCWlzZpJZaVn2azO/nqC4zoLXJLgiQe4g2BItYvrcAvBZWSk/Hg5s1/saLTqh1lZo3TMPtFvbK3XBrjujGbpVnIqIi9oBztDoejx+gEPwAOYagciPM/q191+WNXH8/k6ldFgEJtB8Kc39NpnFG5oBGApIi+8XSucOZ3Vr3T9WSHkh/kYWUHJ4BcPMl8HwTV2/GERFSNB7vYBBBB0LLhCuWCnF0b7RsZBhA40dXU1TPnbFUIdwe+NBXx9gDswJAS74YzXR6xGvbw+UTrQx8VdR7f+6hYqOn2atny1nr79bicdOnSYOlAH+mTVR9Qnsh/Nf+lPVN/YQGfLKsg2agSNGTOG5r/yMtXW1FBG5glauvAtuvEXv6Yf9+yk195eSKOTkygvL59stiQaGBdP736wkspLy2jj+rUUP3Aw5efl0IoVy+mb7TsptGNHeuvVBRTSuw/lniqgysqLZBs9mpYt+RtF9u1P+/btpZ3f76UTOSdpydtv0E0330rz5v4v5Z7Ko5MFxZQQ049WrVpNQ4YkUnFxIf20fw89cP9ESkyxsd2GuK/8sziQAcxs7K8l9GTuMxgpvCiq3QUGtxq2LuM9KoyS4aLC/YIxw4wiNDSUuW5F5QozNyg97ToYv1f81x05RV3Y7n/mTAFVVJSztRa4uSCzWjnmdVvJqai8AJehQ5OopKSQPRtQ+GgXs2I+A/d2P9ypj/cXv7ifRq5k8IvXEzB4QN32DTwKkxXcX6xd8XcIsYYlIi+8XSucOZ3lr+hZS5JOIuANBLx55llTU6OSl5dtyZYoXa/wrqyuhoYG9qXoZe+9o1yoKFfrX75ssfLMU0+y+EsvPq/ccNONSl1tDYvXXL6kXudmZ7I8ZOz67hslokeEcu5ssVJfV6vExSUoKIuw8G+vs69U43rx4jeVPd9vY+lFBaeUpKQkdk7am68tUD75eDlLx7/77ktV9v6ww4lOz49aQFGUp+fMVBa984Y2SV67QEBUXlxUIbO8jICcUVmackngbQQ8cRUZ8YC1LiN3oZ5WlK6qppbNXFD+1lvG0KPTn6Ts40do0uQpVHIWbq5GSoizn+cXREF0153/TaFXT5eAi/LJWU/Q4SOZbGZRkJ2tsjHmppso8uppGQnXxdO4sWNZXkpKMm3fsYtdp6cfod2799OMWf/D4tWXLtPlS9XUqTO2Vzevle3Y8QNlZOaodXM6PT8qwdULb2Gur9cf46Ly4o9999U+SUPlq3fGD/nCojZ3DeFFWLgc4UbyNNhdUNblRem6delMh9LTHdiZmfYMPff0LLpz/ES2seHF+a+q+V3DmtfF3v7r69SvbxR99NGnbN2sa+fm45FCQ5vdlsFBze6u4A7B1HR1y/vXX2+j/T9+T0EdOqhrfxer7N/FuniheT0P71r9uHePaiBVZvDuk4YfbfqopET69y67QdSmy2tjBETlxbi0TG0NBORmitZAVdZpiABeyMRRQjbbGHYyRUuMlGEDrZB4obKSomPsmx7+vrL5U/UKOW4+wDrSgL72rdEf/eMDqnfz5dbbbvsP+nTVx2xxHjv+SkuK2WaLxvoGulhdpfbs9tt/RYvfeUuNHzt8QL02u/j9A4/Qj/sP05ZN61WS3bu20fGMn9W4vJAI+DIC0lD58t2RvF1bBHDaqi68MvdPNOm+B+mGm26k3r2bT8mGqw2L0Dykpc2hf65aTbbRNsrMyqKe4c6nJHBa/qstv3jREtp/MJ1sNhuNSh5F7y1dysjw4bkNGzczl+S+3d+Rnm7Je+/x6hz4UROJqFPnLrRxwxpa+O4yGjlyJKv/3aXvs00hWjp5LRHwVQTk96h89c5IviQCEgGJgESAISBnVFIQJAISAYmARMCnEZCGyqdvj2ROIiARkAhIBKShkjIgEZAISAQkAj6NgDRUPn17JHMSAYmAREAiIA2VlAGJgERAIiAR8GkEpKHy6dsjmZMISAQkAhIBaaikDEgEJAISAYmATyMgDZVP3x7JnERAIiARkAj8P2ISkH713L4qAAAAAElFTkSuQmCC"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.验证曲线\n",
    "和学习曲线不同，验证曲线的横轴为某个超参数的一系列值，由此比较不同超参数设置下的模型准确值。从下图的验证曲线可以看到，随着超参数设置的改变，模型可能会有从欠拟合到合适再到过拟合的过程，进而可以选择一个合适的超参数设置来提高模型的性能。\n",
    "![image.png](attachment:image.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#以Xgboost为例，该网格搜索代码示例如下\n",
    "import xgboost as xgb\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "cancer = load_breast_cancer()\n",
    "x = cancer.data[:50]\n",
    "y = cancer.target[:50]\n",
    "train_x, valid_x, train_y, valid_y = train_test_split(x, y, test_size=0.333, random_state=0)  # 分训练集和验证集\n",
    "# 这里不需要Dmatrix\n",
    "\n",
    "parameters = {\n",
    "    'max_depth': [5, 10, 15, 20, 25],\n",
    "    'learning_rate': [0.01, 0.02, 0.05, 0.1, 0.15],\n",
    "    'n_estimators': [50, 100, 200, 300, 500],\n",
    "    'min_child_weight': [0, 2, 5, 10, 20],\n",
    "    'max_delta_step': [0, 0.2, 0.6, 1, 2],\n",
    "    'subsample': [0.6, 0.7, 0.8, 0.85, 0.95],\n",
    "    'colsample_bytree': [0.5, 0.6, 0.7, 0.8, 0.9],\n",
    "    'reg_alpha': [0, 0.25, 0.5, 0.75, 1],\n",
    "    'reg_lambda': [0.2, 0.4, 0.6, 0.8, 1],\n",
    "    'scale_pos_weight': [0.2, 0.4, 0.6, 0.8, 1]\n",
    "}\n",
    "\n",
    "xlf = xgb.XGBClassifier(max_depth=10,\n",
    "                        learning_rate=0.01,\n",
    "                        n_estimators=2000,\n",
    "                        silent=True,\n",
    "                        objective='binary:logistic',\n",
    "                        nthread=-1,\n",
    "                        gamma=0,\n",
    "                        min_child_weight=1,\n",
    "                        max_delta_step=0,\n",
    "                        subsample=0.85,\n",
    "                        colsample_bytree=0.7,\n",
    "                        colsample_bylevel=1,\n",
    "                        reg_alpha=0,\n",
    "                        reg_lambda=1,\n",
    "                        scale_pos_weight=1,\n",
    "                        seed=1440,\n",
    "                        missing=None)\n",
    "\n",
    "# 有了gridsearch我们便不需要fit函数\n",
    "gsearch = GridSearchCV(xlf, param_grid=parameters, scoring='accuracy', cv=3)\n",
    "gsearch.fit(train_x, train_y)\n",
    "\n",
    "print(\"Best score: %0.3f\" % gsearch.best_score_)\n",
    "print(\"Best parameters set:\")\n",
    "best_parameters = gsearch.best_estimator_.get_params()\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"\\t%s: %r\" % (param_name, best_parameters[param_name]))\n",
    "#极其耗费时间,电脑没执行完"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 智慧海洋数据集模型代码示例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## lightGBM模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from tqdm import tqdm\n",
    "from sklearn.metrics import classification_report, f1_score\n",
    "from sklearn.model_selection import StratifiedKFold, KFold,train_test_split\n",
    "import lightgbm as lgb\n",
    "import os\n",
    "import warnings\n",
    "from hyperopt import fmin, tpe, hp, STATUS_OK, Trials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_df=pd.read_csv('group_df.csv',index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "use_train = all_df[all_df['label'] != -1]\n",
    "use_test = all_df[all_df['label'] == -1]#label为-1时是测试集\n",
    "use_feats = [c for c in use_train.columns if c not in ['ID', 'label']]\n",
    "X_train,X_verify,y_train,y_verify= train_test_split(use_train[use_feats],use_train['label'],test_size=0.3,random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.根据特征的重要性进行特征选择"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\SOFTWEAR_H\\Anaconda3\\lib\\site-packages\\lightgbm\\callback.py:186: UserWarning: Early stopping is not available in dart mode\n",
      "  warnings.warn('Early stopping is not available in dart mode')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total feature best score: defaultdict(<class 'collections.OrderedDict'>, {'valid': OrderedDict([('f1_score', 0.9004541298211368)])})\n",
      "total feature importance: [('pos_neq_zero_speed_q_40', 1783), ('lat_lon_countvec_1_x', 1771), ('rank2_mode_lat', 1737), ('pos_neq_zero_speed_median', 1379), ('pos_neq_zero_speed_q_60', 1369), ('lat_lon_tfidf_0_x', 1251), ('pos_neq_zero_speed_q_80', 1194), ('sample_tfidf_0_x', 1168), ('w2v_9_mean', 1134), ('lat_lon_tfidf_11_x', 963), ('rank3_mode_lat', 946), ('w2v_5_mean', 900), ('w2v_16_mean', 874), ('pos_neq_zero_speed_q_30', 866), ('w2v_12_mean', 862), ('pos_neq_zero_speed_q_70', 856), ('lat_lon_tfidf_9_x', 787), ('grad_tfidf_7_x', 772), ('pos_neq_zero_speed_q_90', 746), ('rank3_mode_cnt', 733), ('grad_tfidf_12_x', 729), ('w2v_4_mean', 697), ('sample_tfidf_14_x', 695), ('lat_lon_tfidf_4_x', 693), ('lat_min', 683), ('w2v_23_mean', 647), ('rank2_mode_lon', 631), ('w2v_26_mean', 626), ('rank1_mode_lon', 620), ('grad_tfidf_15_x', 607), ('speed_neq_zero_speed_q_90', 603), ('grad_tfidf_5_x', 572), ('lat_lon_countvec_22_x', 571), ('lat_lon_countvec_1_y', 565), ('w2v_13_mean', 557), ('w2v_27_mean', 550), ('grad_tfidf_2_x', 507), ('lat_lon_tfidf_20_x', 503), ('lat_lon_countvec_0_x', 499), ('lat_lon_countvec_18_x', 490), ('sample_tfidf_21_x', 488), ('grad_tfidf_14_x', 484), ('lat_lon_countvec_27_x', 470), ('w2v_22_mean', 466), ('lat_lon_tfidf_1_x', 461), ('direction_nunique', 460), ('lon_max', 457), ('w2v_15_mean', 441), ('grad_tfidf_23_x', 431), ('w2v_19_mean', 429), ('w2v_11_mean', 428), ('lat_lon_tfidf_29_x', 420), ('pos_neq_zero_lon_q_10', 417), ('w2v_3_mean', 411), ('lat_lon_tfidf_0_y', 407), ('sample_tfidf_29_x', 406), ('anchor_cnt', 404), ('grad_tfidf_8_x', 397), ('sample_tfidf_10_x', 397), ('sample_tfidf_12_x', 385), ('w2v_28_mean', 384), ('grad_tfidf_13_x', 381), ('direction_q_90', 380), ('speed_neq_zero_lon_min', 374), ('w2v_25_mean', 371), ('anchor_ratio', 367), ('lat_lon_tfidf_16_x', 367), ('rank1_mode_lat', 365), ('w2v_18_mean', 365), ('sample_tfidf_23_x', 364), ('lon_min', 354), ('grad_tfidf_0_x', 351), ('pos_neq_zero_lat_q_90', 341), ('w2v_20_mean', 341), ('sample_tfidf_4_x', 334), ('lat_lon_tfidf_23_x', 332), ('sample_tfidf_0_y', 328), ('pos_neq_zero_direction_q_90', 326), ('speed_neq_zero_direction_nunique', 326), ('sample_tfidf_19_x', 323), ('lat_lon_countvec_9_x', 319), ('pos_neq_zero_lon_q_90', 314), ('w2v_8_mean', 312), ('grad_tfidf_3_x', 309), ('lon_median', 305), ('pos_neq_zero_speed_q_20', 304), ('lat_lon_countvec_4_x', 304), ('lat_mean', 301), ('speed_neq_zero_lon_max', 301), ('lat_lon_tfidf_14_x', 301), ('speed_neq_zero_lat_min', 300), ('lat_lon_countvec_5_x', 296), ('speed_neq_zero_speed_q_80', 294), ('grad_tfidf_16_x', 293), ('rank3_mode_lon', 292), ('lat_lon_tfidf_18_x', 291), ('w2v_7_mean', 290), ('grad_tfidf_6_x', 285), ('grad_tfidf_20_x', 283), ('grad_tfidf_18_x', 282), ('w2v_0_mean', 280), ('grad_tfidf_21_x', 279), ('grad_tfidf_22_x', 273), ('sample_tfidf_24_x', 273), ('speed_q_90', 271), ('w2v_2_mean', 271), ('lat_max', 264), ('sample_tfidf_9_x', 264), ('grad_tfidf_11_x', 262), ('lon_q_20', 260), ('rank1_mode_cnt', 258), ('speed_max', 256), ('lat_lon_tfidf_12_x', 251), ('pos_neq_zero_lon_q_20', 248), ('lat_lon_tfidf_28_x', 242), ('speed_neq_zero_direction_q_60', 241), ('sample_tfidf_11_x', 241), ('w2v_17_mean', 241), ('sample_tfidf_13_x', 238), ('w2v_14_mean', 236), ('lat_nunique', 235), ('grad_tfidf_4_x', 234), ('w2v_21_mean', 234), ('sample_tfidf_5_x', 231), ('lat_lon_tfidf_9_y', 225), ('speed_neq_zero_lat_q_90', 222), ('direction_median', 221), ('sample_tfidf_17_x', 220), ('sample_tfidf_14_y', 216), ('lat_lon_tfidf_21_x', 215), ('lon_q_10', 214), ('lat_lon_tfidf_22_x', 214), ('grad_tfidf_26_x', 213), ('grad_tfidf_7_y', 213), ('w2v_29_mean', 212), ('pos_neq_zero_lat_q_80', 210), ('cnt', 209), ('lat_lon_tfidf_4_y', 208), ('direction_q_60', 204), ('sample_tfidf_18_x', 203), ('lat_lon_tfidf_11_y', 203), ('pos_neq_zero_lat_min', 202), ('pos_neq_zero_speed_mean', 201), ('speed_neq_zero_lat_q_70', 200), ('grad_tfidf_12_y', 198), ('sample_tfidf_20_x', 197), ('w2v_1_mean', 194), ('speed_neq_zero_lat_q_40', 193), ('pos_neq_zero_speed_max', 192), ('grad_tfidf_27_x', 192), ('grad_tfidf_15_y', 191), ('lat_lon_tfidf_19_x', 189), ('lat_median', 187), ('lat_lon_tfidf_15_x', 187), ('lat_q_20', 186), ('lat_q_70', 186), ('lon_q_70', 185), ('w2v_24_mean', 184), ('pos_neq_zero_lat_q_40', 183), ('grad_tfidf_25_x', 181), ('w2v_10_mean', 181), ('lon_mean', 180), ('sample_tfidf_27_x', 180), ('w2v_6_mean', 180), ('lat_lon_tfidf_24_x', 178), ('lat_lon_countvec_12_x', 178), ('pos_neq_zero_lat_mean', 177), ('speed_neq_zero_speed_q_70', 174), ('speed_neq_zero_direction_q_80', 172), ('rank2_mode_cnt', 172), ('speed_neq_zero_lat_nunique', 171), ('lat_lon_tfidf_2_x', 171), ('sample_tfidf_25_x', 170), ('lat_lon_tfidf_5_x', 169), ('lat_lon_countvec_26_x', 167), ('grad_tfidf_9_x', 166), ('lat_lon_countvec_28_x', 163), ('lat_lon_countvec_22_y', 163), ('sample_tfidf_1_x', 162), ('pos_neq_zero_direction_nunique', 161), ('pos_neq_zero_speed_q_10', 157), ('sample_tfidf_16_x', 155), ('speed_neq_zero_direction_q_90', 154), ('grad_tfidf_14_y', 153), ('lat_lon_tfidf_7_x', 151), ('pos_neq_zero_direction_q_80', 149), ('lat_q_80', 148), ('grad_tfidf_23_y', 148), ('lat_lon_countvec_11_x', 147), ('sample_tfidf_22_x', 146), ('speed_neq_zero_lat_max', 144), ('sample_tfidf_15_x', 144), ('grad_tfidf_2_y', 144), ('pos_neq_zero_lat_q_10', 142), ('lat_lon_tfidf_1_y', 142), ('lat_lon_countvec_16_x', 141), ('grad_tfidf_13_y', 138), ('lat_lon_countvec_29_x', 136), ('lat_lon_tfidf_29_y', 136), ('grad_tfidf_5_y', 136), ('direction_max', 135), ('pos_neq_zero_lon_median', 134), ('lat_lon_tfidf_27_x', 134), ('lon_q_80', 133), ('lat_lon_countvec_15_x', 133), ('pos_neq_zero_lon_max', 132), ('lat_lon_countvec_14_x', 132), ('lat_lon_tfidf_26_x', 131), ('grad_tfidf_19_x', 131), ('sample_tfidf_8_x', 131), ('lat_q_60', 130), ('sample_tfidf_28_x', 130), ('lat_lon_countvec_27_y', 130), ('lat_lon_countvec_6_x', 128), ('lat_lon_countvec_0_y', 128), ('sample_tfidf_12_y', 127), ('lat_lon_tfidf_8_x', 126), ('sample_tfidf_29_y', 126), ('lat_lon_countvec_17_x', 125), ('direction_q_70', 124), ('lat_lon_tfidf_20_y', 124), ('lat_lon_tfidf_3_x', 121), ('sample_tfidf_21_y', 120), ('grad_tfidf_0_y', 119), ('pos_neq_zero_lat_median', 118), ('lat_lon_tfidf_16_y', 118), ('grad_tfidf_10_x', 117), ('sample_tfidf_2_x', 116), ('lat_lon_countvec_4_y', 116), ('speed_median', 115), ('pos_neq_zero_direction_q_10', 115), ('speed_neq_zero_lon_mean', 115), ('pos_neq_zero_direction_max', 114), ('lat_q_40', 113), ('grad_tfidf_1_x', 113), ('speed_nunique', 111), ('sample_tfidf_23_y', 111), ('speed_q_30', 110), ('pos_neq_zero_lat_q_30', 110), ('lat_lon_tfidf_10_x', 110), ('lat_lon_countvec_10_x', 110), ('lat_lon_tfidf_23_y', 109), ('pos_neq_zero_speed_min', 106), ('speed_neq_zero_lat_q_60', 106), ('lat_lon_countvec_21_x', 106), ('lat_lon_countvec_18_y', 106), ('lat_lon_tfidf_17_x', 105), ('grad_tfidf_8_y', 103), ('grad_tfidf_6_y', 102), ('sample_tfidf_10_y', 101), ('pos_neq_zero_lon_min', 100), ('lat_lon_countvec_8_x', 100), ('lat_lon_countvec_9_y', 100), ('direction_mean', 99), ('grad_tfidf_21_y', 99), ('lat_lon_tfidf_6_x', 98), ('lat_lon_tfidf_18_y', 97), ('direction_q_80', 96), ('pos_neq_zero_direction_q_70', 96), ('lat_lon_countvec_20_x', 95), ('speed_neq_zero_direction_q_70', 93), ('lat_lon_countvec_25_x', 93), ('lat_lon_countvec_23_x', 92), ('lat_lon_tfidf_14_y', 92), ('lat_q_90', 91), ('sample_tfidf_7_x', 91), ('pos_neq_zero_lon_q_70', 90), ('lat_lon_countvec_5_y', 90), ('pos_neq_zero_direction_q_20', 89), ('lat_lon_tfidf_12_y', 89), ('lat_lon_tfidf_28_y', 89), ('sample_tfidf_4_y', 89), ('direction_q_40', 88), ('pos_neq_zero_lat_q_20', 87), ('grad_tfidf_17_x', 87), ('sample_tfidf_9_y', 87), ('sample_tfidf_24_y', 87), ('pos_neq_zero_lat_max', 86), ('pos_neq_zero_lon_mean', 86), ('speed_neq_zero_direction_q_40', 86), ('lat_lon_countvec_7_x', 86), ('speed_neq_zero_speed_q_40', 85), ('sample_tfidf_6_x', 84), ('sample_tfidf_19_y', 84), ('speed_min', 83), ('direction_q_10', 83), ('lat_lon_countvec_19_x', 83), ('grad_tfidf_24_x', 83), ('speed_q_60', 82), ('lat_lon_tfidf_25_x', 82), ('sample_tfidf_3_x', 82), ('grad_tfidf_22_y', 82), ('direction_q_30', 80), ('speed_neq_zero_direction_mean', 80), ('grad_tfidf_18_y', 77), ('lat_q_10', 76), ('speed_neq_zero_speed_max', 75), ('grad_tfidf_3_y', 75), ('sample_tfidf_11_y', 75), ('lon_nunique', 74), ('lon_q_90', 74), ('speed_neq_zero_lon_q_10', 74), ('speed_neq_zero_speed_median', 74), ('grad_tfidf_28_x', 74), ('grad_tfidf_20_y', 74), ('speed_neq_zero_lon_q_70', 73), ('lat_lon_tfidf_24_y', 73), ('pos_neq_zero_lat_q_60', 72), ('lat_lon_countvec_2_x', 72), ('lat_lon_countvec_3_x', 69), ('sample_tfidf_20_y', 69), ('lat_lon_tfidf_13_x', 68), ('grad_tfidf_16_y', 68), ('sample_tfidf_13_y', 67), ('speed_neq_zero_lon_q_30', 66), ('speed_q_40', 65), ('grad_tfidf_4_y', 65), ('sample_tfidf_5_y', 65), ('lat_q_30', 64), ('pos_neq_zero_direction_median', 64), ('speed_neq_zero_lat_median', 64), ('grad_tfidf_11_y', 64), ('grad_tfidf_27_y', 64), ('lat_lon_tfidf_19_y', 62), ('pos_neq_zero_lon_q_40', 61), ('lat_lon_countvec_26_y', 61), ('pos_neq_zero_lon_q_80', 60), ('sample_tfidf_17_y', 60), ('lon_q_40', 59), ('lat_lon_countvec_28_y', 59), ('lat_lon_tfidf_22_y', 57), ('grad_tfidf_29_x', 56), ('lat_lon_countvec_12_y', 56), ('sample_tfidf_15_y', 56), ('sample_tfidf_27_y', 56), ('speed_q_70', 55), ('lat_lon_tfidf_21_y', 55), ('grad_tfidf_9_y', 55), ('sample_tfidf_25_y', 55), ('pos_neq_zero_direction_mean', 54), ('sample_tfidf_26_x', 54), ('sample_tfidf_18_y', 53), ('speed_neq_zero_lon_q_90', 51), ('speed_neq_zero_direction_max', 51), ('lat_lon_tfidf_5_y', 50), ('pos_neq_zero_direction_q_60', 49), ('sample_tfidf_2_y', 49), ('pos_neq_zero_lon_q_60', 48), ('speed_neq_zero_speed_mean', 48), ('lat_lon_tfidf_15_y', 48), ('pos_neq_zero_direction_q_30', 47), ('speed_neq_zero_lon_nunique', 47), ('lat_lon_countvec_24_x', 47), ('sample_tfidf_8_y', 47), ('lat_lon_tfidf_10_y', 46), ('lon_q_60', 45), ('pos_neq_zero_lat_q_70', 45), ('speed_neq_zero_direction_q_10', 45), ('lat_lon_tfidf_3_y', 45), ('speed_neq_zero_lat_mean', 43), ('speed_neq_zero_lat_q_80', 43), ('lat_lon_tfidf_2_y', 43), ('lat_lon_tfidf_8_y', 43), ('grad_tfidf_19_y', 43), ('grad_tfidf_25_y', 43), ('grad_tfidf_26_y', 43), ('lon_q_30', 42), ('speed_neq_zero_lon_q_20', 42), ('pos_neq_zero_speed_nunique', 41), ('speed_neq_zero_speed_nunique', 41), ('speed_neq_zero_speed_q_30', 41), ('lat_lon_tfidf_7_y', 41), ('lat_lon_tfidf_17_y', 41), ('lat_lon_countvec_14_y', 41), ('grad_tfidf_10_y', 41), ('lat_lon_tfidf_26_y', 40), ('grad_tfidf_1_y', 40), ('speed_neq_zero_lat_q_20', 39), ('speed_q_80', 38), ('speed_neq_zero_lat_q_30', 38), ('lat_lon_countvec_15_y', 38), ('pos_neq_zero_direction_q_40', 37), ('speed_neq_zero_direction_median', 37), ('pos_neq_zero_lon_q_30', 36), ('lat_lon_countvec_11_y', 36), ('lat_lon_countvec_21_y', 35), ('sample_tfidf_28_y', 35), ('speed_neq_zero_speed_q_60', 34), ('lat_lon_countvec_29_y', 34), ('sample_tfidf_1_y', 34), ('sample_tfidf_22_y', 34), ('lat_lon_countvec_6_y', 33), ('lat_lon_countvec_10_y', 33), ('lat_lon_countvec_16_y', 33), ('speed_mean', 32), ('lat_lon_countvec_17_y', 31), ('lat_lon_countvec_23_y', 31), ('speed_neq_zero_direction_q_30', 30), ('lat_lon_tfidf_13_y', 30), ('sample_tfidf_16_y', 30), ('speed_neq_zero_lat_q_10', 29), ('lat_lon_tfidf_27_y', 29), ('grad_tfidf_17_y', 29), ('lat_lon_countvec_13_x', 27), ('lat_lon_countvec_19_y', 27), ('grad_tfidf_24_y', 26), ('speed_neq_zero_lon_q_40', 25), ('lat_lon_tfidf_25_y', 25), ('lat_lon_countvec_8_y', 25), ('speed_neq_zero_lon_median', 24), ('speed_neq_zero_speed_min', 24), ('lat_lon_countvec_25_y', 24), ('sample_tfidf_6_y', 24), ('pos_neq_zero_lat_nunique', 23), ('speed_neq_zero_lon_q_80', 23), ('lat_lon_countvec_20_y', 23), ('speed_neq_zero_speed_q_10', 22), ('lat_lon_countvec_3_y', 22), ('grad_tfidf_28_y', 22), ('sample_tfidf_7_y', 22), ('lat_lon_countvec_7_y', 21), ('sample_tfidf_26_y', 21), ('lat_lon_tfidf_6_y', 20), ('sample_tfidf_3_y', 20), ('grad_tfidf_29_y', 18), ('speed_neq_zero_lon_q_60', 16), ('speed_neq_zero_speed_q_20', 14), ('lat_lon_countvec_24_y', 14), ('lat_lon_countvec_2_y', 11), ('speed_neq_zero_direction_q_20', 9), ('lat_lon_countvec_13_y', 8), ('speed_q_10', 7), ('pos_neq_zero_lon_nunique', 5), ('direction_q_20', 4), ('speed_q_20', 2), ('pos_neq_zero_direction_min', 2), ('direction_min', 0), ('speed_neq_zero_direction_min', 0)]\n",
      "select forward 200 features:[('pos_neq_zero_speed_q_40', 1783), ('lat_lon_countvec_1_x', 1771), ('rank2_mode_lat', 1737), ('pos_neq_zero_speed_median', 1379), ('pos_neq_zero_speed_q_60', 1369), ('lat_lon_tfidf_0_x', 1251), ('pos_neq_zero_speed_q_80', 1194), ('sample_tfidf_0_x', 1168), ('w2v_9_mean', 1134), ('lat_lon_tfidf_11_x', 963), ('rank3_mode_lat', 946), ('w2v_5_mean', 900), ('w2v_16_mean', 874), ('pos_neq_zero_speed_q_30', 866), ('w2v_12_mean', 862), ('pos_neq_zero_speed_q_70', 856), ('lat_lon_tfidf_9_x', 787), ('grad_tfidf_7_x', 772), ('pos_neq_zero_speed_q_90', 746), ('rank3_mode_cnt', 733), ('grad_tfidf_12_x', 729), ('w2v_4_mean', 697), ('sample_tfidf_14_x', 695), ('lat_lon_tfidf_4_x', 693), ('lat_min', 683), ('w2v_23_mean', 647), ('rank2_mode_lon', 631), ('w2v_26_mean', 626), ('rank1_mode_lon', 620), ('grad_tfidf_15_x', 607), ('speed_neq_zero_speed_q_90', 603), ('grad_tfidf_5_x', 572), ('lat_lon_countvec_22_x', 571), ('lat_lon_countvec_1_y', 565), ('w2v_13_mean', 557), ('w2v_27_mean', 550), ('grad_tfidf_2_x', 507), ('lat_lon_tfidf_20_x', 503), ('lat_lon_countvec_0_x', 499), ('lat_lon_countvec_18_x', 490), ('sample_tfidf_21_x', 488), ('grad_tfidf_14_x', 484), ('lat_lon_countvec_27_x', 470), ('w2v_22_mean', 466), ('lat_lon_tfidf_1_x', 461), ('direction_nunique', 460), ('lon_max', 457), ('w2v_15_mean', 441), ('grad_tfidf_23_x', 431), ('w2v_19_mean', 429), ('w2v_11_mean', 428), ('lat_lon_tfidf_29_x', 420), ('pos_neq_zero_lon_q_10', 417), ('w2v_3_mean', 411), ('lat_lon_tfidf_0_y', 407), ('sample_tfidf_29_x', 406), ('anchor_cnt', 404), ('grad_tfidf_8_x', 397), ('sample_tfidf_10_x', 397), ('sample_tfidf_12_x', 385), ('w2v_28_mean', 384), ('grad_tfidf_13_x', 381), ('direction_q_90', 380), ('speed_neq_zero_lon_min', 374), ('w2v_25_mean', 371), ('anchor_ratio', 367), ('lat_lon_tfidf_16_x', 367), ('rank1_mode_lat', 365), ('w2v_18_mean', 365), ('sample_tfidf_23_x', 364), ('lon_min', 354), ('grad_tfidf_0_x', 351), ('pos_neq_zero_lat_q_90', 341), ('w2v_20_mean', 341), ('sample_tfidf_4_x', 334), ('lat_lon_tfidf_23_x', 332), ('sample_tfidf_0_y', 328), ('pos_neq_zero_direction_q_90', 326), ('speed_neq_zero_direction_nunique', 326), ('sample_tfidf_19_x', 323), ('lat_lon_countvec_9_x', 319), ('pos_neq_zero_lon_q_90', 314), ('w2v_8_mean', 312), ('grad_tfidf_3_x', 309), ('lon_median', 305), ('pos_neq_zero_speed_q_20', 304), ('lat_lon_countvec_4_x', 304), ('lat_mean', 301), ('speed_neq_zero_lon_max', 301), ('lat_lon_tfidf_14_x', 301), ('speed_neq_zero_lat_min', 300), ('lat_lon_countvec_5_x', 296), ('speed_neq_zero_speed_q_80', 294), ('grad_tfidf_16_x', 293), ('rank3_mode_lon', 292), ('lat_lon_tfidf_18_x', 291), ('w2v_7_mean', 290), ('grad_tfidf_6_x', 285), ('grad_tfidf_20_x', 283), ('grad_tfidf_18_x', 282), ('w2v_0_mean', 280), ('grad_tfidf_21_x', 279), ('grad_tfidf_22_x', 273), ('sample_tfidf_24_x', 273), ('speed_q_90', 271), ('w2v_2_mean', 271), ('lat_max', 264), ('sample_tfidf_9_x', 264), ('grad_tfidf_11_x', 262), ('lon_q_20', 260), ('rank1_mode_cnt', 258), ('speed_max', 256), ('lat_lon_tfidf_12_x', 251), ('pos_neq_zero_lon_q_20', 248), ('lat_lon_tfidf_28_x', 242), ('speed_neq_zero_direction_q_60', 241), ('sample_tfidf_11_x', 241), ('w2v_17_mean', 241), ('sample_tfidf_13_x', 238), ('w2v_14_mean', 236), ('lat_nunique', 235), ('grad_tfidf_4_x', 234), ('w2v_21_mean', 234), ('sample_tfidf_5_x', 231), ('lat_lon_tfidf_9_y', 225), ('speed_neq_zero_lat_q_90', 222), ('direction_median', 221), ('sample_tfidf_17_x', 220), ('sample_tfidf_14_y', 216), ('lat_lon_tfidf_21_x', 215), ('lon_q_10', 214), ('lat_lon_tfidf_22_x', 214), ('grad_tfidf_26_x', 213), ('grad_tfidf_7_y', 213), ('w2v_29_mean', 212), ('pos_neq_zero_lat_q_80', 210), ('cnt', 209), ('lat_lon_tfidf_4_y', 208), ('direction_q_60', 204), ('sample_tfidf_18_x', 203), ('lat_lon_tfidf_11_y', 203), ('pos_neq_zero_lat_min', 202), ('pos_neq_zero_speed_mean', 201), ('speed_neq_zero_lat_q_70', 200), ('grad_tfidf_12_y', 198), ('sample_tfidf_20_x', 197), ('w2v_1_mean', 194), ('speed_neq_zero_lat_q_40', 193), ('pos_neq_zero_speed_max', 192), ('grad_tfidf_27_x', 192), ('grad_tfidf_15_y', 191), ('lat_lon_tfidf_19_x', 189), ('lat_median', 187), ('lat_lon_tfidf_15_x', 187), ('lat_q_20', 186), ('lat_q_70', 186), ('lon_q_70', 185), ('w2v_24_mean', 184), ('pos_neq_zero_lat_q_40', 183), ('grad_tfidf_25_x', 181), ('w2v_10_mean', 181), ('lon_mean', 180), ('sample_tfidf_27_x', 180), ('w2v_6_mean', 180), ('lat_lon_tfidf_24_x', 178), ('lat_lon_countvec_12_x', 178), ('pos_neq_zero_lat_mean', 177), ('speed_neq_zero_speed_q_70', 174), ('speed_neq_zero_direction_q_80', 172), ('rank2_mode_cnt', 172), ('speed_neq_zero_lat_nunique', 171), ('lat_lon_tfidf_2_x', 171), ('sample_tfidf_25_x', 170), ('lat_lon_tfidf_5_x', 169), ('lat_lon_countvec_26_x', 167), ('grad_tfidf_9_x', 166), ('lat_lon_countvec_28_x', 163), ('lat_lon_countvec_22_y', 163), ('sample_tfidf_1_x', 162), ('pos_neq_zero_direction_nunique', 161), ('pos_neq_zero_speed_q_10', 157), ('sample_tfidf_16_x', 155), ('speed_neq_zero_direction_q_90', 154), ('grad_tfidf_14_y', 153), ('lat_lon_tfidf_7_x', 151), ('pos_neq_zero_direction_q_80', 149), ('lat_q_80', 148), ('grad_tfidf_23_y', 148), ('lat_lon_countvec_11_x', 147), ('sample_tfidf_22_x', 146), ('speed_neq_zero_lat_max', 144), ('sample_tfidf_15_x', 144), ('grad_tfidf_2_y', 144), ('pos_neq_zero_lat_q_10', 142), ('lat_lon_tfidf_1_y', 142), ('lat_lon_countvec_16_x', 141), ('grad_tfidf_13_y', 138), ('lat_lon_countvec_29_x', 136), ('lat_lon_tfidf_29_y', 136), ('grad_tfidf_5_y', 136)]\n"
     ]
    }
   ],
   "source": [
    "##############特征选择参数###################\n",
    "selectFeatures = 200 # 控制特征数\n",
    "earlyStopping = 100 # 控制早停\n",
    "select_num_boost_round = 1000 # 特征选择训练轮次\n",
    "#首先设置基础参数\n",
    "selfParam = {\n",
    "    'learning_rate':0.01, # 学习率\n",
    "    'boosting':'dart', # 算法类型, gbdt,dart\n",
    "    'objective':'multiclass', # 多分类\n",
    "    'metric':'None',\n",
    "    'num_leaves':32, # \n",
    "    'feature_fraction':0.7, # 训练特征比例\n",
    "    'bagging_fraction':0.8, # 训练样本比例 \n",
    "    'min_data_in_leaf':30, # 叶子最小样本\n",
    "    'num_class': 3,\n",
    "    'max_depth':6, # 树的最大深度\n",
    "    \n",
    "    'num_threads':8,#LightGBM 的线程数\n",
    "    'min_data_in_bin':30, # 单箱数据量\n",
    "    'max_bin':256, # 最大分箱数 \n",
    "    'is_unbalance':True, # 非平衡样本\n",
    "    'train_metric':True,\n",
    "    'verbose':-1,\n",
    "}\n",
    "# 特征选择 ---------------------------------------------------------------------------------\n",
    "def f1_score_eval(preds, valid_df):\n",
    "    labels = valid_df.get_label()\n",
    "    preds = np.argmax(preds.reshape(3, -1), axis=0)\n",
    "    scores = f1_score(y_true=labels, y_pred=preds, average='macro')\n",
    "    return 'f1_score', scores, True\n",
    "\n",
    "train_data = lgb.Dataset(data=X_train,label=y_train,feature_name=use_feats)\n",
    "valid_data = lgb.Dataset(data=X_verify,label=y_verify,reference=train_data,feature_name=use_feats)\n",
    "\n",
    "sm = lgb.train(params=selfParam,train_set=train_data,num_boost_round=select_num_boost_round,\n",
    "                      valid_sets=[valid_data],valid_names=['valid'],\n",
    "                      feature_name=use_feats,\n",
    "                      early_stopping_rounds=earlyStopping,verbose_eval=False,keep_training_booster=True,feval=f1_score_eval)\n",
    "features_importance = {k:v for k,v in zip(sm.feature_name(),sm.feature_importance(iteration=sm.best_iteration))}\n",
    "sort_feature_importance = sorted(features_importance.items(),key=lambda x:x[1],reverse=True)\n",
    "print('total feature best score:', sm.best_score)\n",
    "print('total feature importance:',sort_feature_importance)\n",
    "print('select forward {} features:{}'.format(selectFeatures,sort_feature_importance[:selectFeatures]))\n",
    "#model_feature是选择的超参数\n",
    "model_feature = [k[0] for k in sort_feature_importance[:selectFeatures]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[贝叶斯优化介绍](https://github.com/FontTian/hyperopt-doc-zh/wiki/FMin)也是在建模调参过程中常用的一种方法，下面是通过贝叶斯优化进行超参数选择的代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练集f1_score:1.0,测试集f1_score:0.9238060849905194,loss_f1_score:0.07619391500948058                                       \n",
      "训练集f1_score:0.9414337502771342,测试集f1_score:0.8878751759836653,loss_f1_score:0.11212482401633472                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9275451088133652,loss_f1_score:0.07245489118663484                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9262405937033683,loss_f1_score:0.07375940629663169                                       \n",
      "训练集f1_score:0.9708237804866381,测试集f1_score:0.9105982243190386,loss_f1_score:0.08940177568096142                        \n",
      "训练集f1_score:0.9689912364726484,测试集f1_score:0.9086459359345839,loss_f1_score:0.09135406406541613                        \n",
      "训练集f1_score:0.9841597696688008,测试集f1_score:0.9027075194168233,loss_f1_score:0.09729248058317674                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9215512877825286,loss_f1_score:0.0784487122174714                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.924555451978199,loss_f1_score:0.075444548021801                                          \n",
      "训练集f1_score:0.998357894114157,测试集f1_score:0.9157797895654226,loss_f1_score:0.08422021043457739                         \n",
      "训练集f1_score:1.0,测试集f1_score:0.9225868784774544,loss_f1_score:0.07741312152254565                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9188521505717673,loss_f1_score:0.08114784942823272                                       \n",
      "训练集f1_score:0.9268245763808158,测试集f1_score:0.8763935795977332,loss_f1_score:0.12360642040226677                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9215959099478135,loss_f1_score:0.07840409005218651                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9265015559936258,loss_f1_score:0.07349844400637418                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9143628354188641,loss_f1_score:0.0856371645811359                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9202754009210264,loss_f1_score:0.07972459907897356                                       \n",
      "训练集f1_score:0.9550283459834631,测试集f1_score:0.8923546584333147,loss_f1_score:0.10764534156668526                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9255732985564632,loss_f1_score:0.0744267014435368                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.926093875740129,loss_f1_score:0.07390612425987098                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9275189170142104,loss_f1_score:0.07248108298578959                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9257895202231272,loss_f1_score:0.07421047977687278                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9248738969479765,loss_f1_score:0.0751261030520235                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9272520229049039,loss_f1_score:0.07274797709509606                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9256769527801775,loss_f1_score:0.07432304721982252                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9252959646692677,loss_f1_score:0.07470403533073233                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9280536344383128,loss_f1_score:0.07194636556168721                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9316114105930104,loss_f1_score:0.06838858940698955                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9282603014798921,loss_f1_score:0.07173969852010786                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9169851848129301,loss_f1_score:0.08301481518706988                                       \n",
      "训练集f1_score:0.9998006409358186,测试集f1_score:0.9170084634982812,loss_f1_score:0.08299153650171875                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.919142326688697,loss_f1_score:0.080857673311303                                          \n",
      "训练集f1_score:1.0,测试集f1_score:0.927350422658861,loss_f1_score:0.07264957734113897                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9248086877712395,loss_f1_score:0.07519131222876052                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9170626453496801,loss_f1_score:0.08293735465031993                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9277641923766077,loss_f1_score:0.07223580762339232                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9221988666312404,loss_f1_score:0.0778011333687596                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9225220095934339,loss_f1_score:0.07747799040656611                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9239565521812777,loss_f1_score:0.0760434478187223                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9276828960144917,loss_f1_score:0.07231710398550828                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9205931627810685,loss_f1_score:0.07940683721893149                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9262928923256212,loss_f1_score:0.07370710767437882                                       \n",
      "训练集f1_score:0.9944566925965641,测试集f1_score:0.9103100448505551,loss_f1_score:0.08968995514944489                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9267901922541096,loss_f1_score:0.07320980774589037                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.920503002249437,loss_f1_score:0.07949699775056296                                        \n",
      "训练集f1_score:0.9315809154440894,测试集f1_score:0.888114739372245,loss_f1_score:0.11188526062775495                         \n",
      "训练集f1_score:1.0,测试集f1_score:0.9312944518110373,loss_f1_score:0.06870554818896268                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9303459748533314,loss_f1_score:0.06965402514666863                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.931353840440614,loss_f1_score:0.06864615955938602                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9229280238009058,loss_f1_score:0.07707197619909423                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9081707271979852,loss_f1_score:0.0918292728020148                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9263682433473132,loss_f1_score:0.07363175665268684                                       \n",
      "训练集f1_score:0.9979810910594639,测试集f1_score:0.9137152734108268,loss_f1_score:0.08628472658917319                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9258220879299731,loss_f1_score:0.07417791207002689                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9174454505221505,loss_f1_score:0.08255454947784946                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9271364668867941,loss_f1_score:0.07286353311320592                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9147023183361269,loss_f1_score:0.08529768166387308                                       \n",
      "训练集f1_score:0.9818127606280159,测试集f1_score:0.9017199309349478,loss_f1_score:0.09828006906505216                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9144702886766378,loss_f1_score:0.08552971132336218                                       \n",
      "训练集f1_score:0.9987361493711533,测试集f1_score:0.9152462742627984,loss_f1_score:0.08475372573720164                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9283825864164065,loss_f1_score:0.07161741358359353                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9185245776900096,loss_f1_score:0.08147542230999039                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9176200948292667,loss_f1_score:0.08237990517073335                                       \n",
      "训练集f1_score:0.9993129514194335,测试集f1_score:0.9174352830766729,loss_f1_score:0.08256471692332712                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9276704131051788,loss_f1_score:0.07232958689482116                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9268048760558437,loss_f1_score:0.07319512394415628                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9304568955332027,loss_f1_score:0.06954310446679735                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9222607611550148,loss_f1_score:0.07773923884498524                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9303686983620825,loss_f1_score:0.06963130163791753                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9275281467065163,loss_f1_score:0.07247185329348371                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9263494542572851,loss_f1_score:0.0736505457427149                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9262464202510822,loss_f1_score:0.07375357974891783                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9213298706249988,loss_f1_score:0.07867012937500117                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9255381820063792,loss_f1_score:0.07446181799362084                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9262492441399471,loss_f1_score:0.07375075586005286                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9267529385979496,loss_f1_score:0.0732470614020504                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9279362552557409,loss_f1_score:0.07206374474425914                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9105496558898486,loss_f1_score:0.0894503441101514                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9255677088759965,loss_f1_score:0.07443229112400351                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9258810998636311,loss_f1_score:0.0741189001363689                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9236045683410877,loss_f1_score:0.07639543165891227                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9236482035413927,loss_f1_score:0.07635179645860735                                       \n",
      "训练集f1_score:0.9998006409358186,测试集f1_score:0.9161826380576955,loss_f1_score:0.08381736194230449                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9226427795765888,loss_f1_score:0.0773572204234112                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9227047668043812,loss_f1_score:0.07729523319561882                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9255689533534145,loss_f1_score:0.07443104664658551                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9322007348532765,loss_f1_score:0.06779926514672352                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9169573599775939,loss_f1_score:0.08304264002240613                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9230059720988804,loss_f1_score:0.07699402790111964                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.922697478395862,loss_f1_score:0.07730252160413797                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9079606352786754,loss_f1_score:0.09203936472132457                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9229248123974857,loss_f1_score:0.0770751876025143                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.923913432252704,loss_f1_score:0.07608656774729605                                        \n",
      "训练集f1_score:1.0,测试集f1_score:0.9257200990324236,loss_f1_score:0.07427990096757642                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9276995504041144,loss_f1_score:0.07230044959588555                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9251348482525271,loss_f1_score:0.07486515174747288                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9231090610362633,loss_f1_score:0.07689093896373667                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9164413618677342,loss_f1_score:0.08355863813226583                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9293008018695311,loss_f1_score:0.07069919813046888                                       \n",
      "训练集f1_score:1.0,测试集f1_score:0.9301285411934597,loss_f1_score:0.06987145880654033                                       \n",
      "100%|█████████████████████████████████████████████| 100/100 [33:56<00:00, 20.36s/trial, best loss: 0.06779926514672352]\n",
      "Search best param: {'bagging_fraction': 0.7310343530671259, 'boosting': 'gbdt', 'feature_fraction': 0.8644701162989126, 'learning_rate': 0.0483933201073737, 'min_data_in_leaf': 15, 'num_boost_round': 1100, 'num_leaves': 60, 'objective': 'multiclass', 'max_depth': 7, 'num_threads': 8, 'is_unbalance': True, 'metric': 'None', 'train_metric': True, 'verbose': -1, 'bagging_freq': 5, 'num_class': 3, 'feature_pre_filter': False}\n"
     ]
    }
   ],
   "source": [
    "##############超参数优化的超参域###################\n",
    "spaceParam = {\n",
    "    'boosting': hp.choice('boosting',['gbdt','dart']),\n",
    "    'learning_rate':hp.loguniform('learning_rate', np.log(0.01), np.log(0.05)),\n",
    "    'num_leaves': hp.quniform('num_leaves', 3, 66, 3), \n",
    "    'feature_fraction': hp.uniform('feature_fraction', 0.7,1),\n",
    "    'min_data_in_leaf': hp.quniform('min_data_in_leaf', 10, 50,5), \n",
    "    'num_boost_round':hp.quniform('num_boost_round',500,2000,100), \n",
    "    'bagging_fraction':hp.uniform('bagging_fraction',0.6,1)  \n",
    "}\n",
    "# 超参数优化 ---------------------------------------------------------------------------------\n",
    "def getParam(param):\n",
    "    for k in ['num_leaves', 'min_data_in_leaf','num_boost_round']:\n",
    "        param[k] = int(float(param[k]))\n",
    "    for k in ['learning_rate', 'feature_fraction','bagging_fraction']:\n",
    "        param[k] = float(param[k])\n",
    "    if param['boosting'] == 0:\n",
    "        param['boosting'] = 'gbdt'\n",
    "    elif param['boosting'] == 1:\n",
    "        param['boosting'] = 'dart'\n",
    "    # 添加固定参数\n",
    "    param['objective'] = 'multiclass'\n",
    "    param['max_depth'] = 7\n",
    "    param['num_threads'] = 8\n",
    "    param['is_unbalance'] = True\n",
    "    param['metric'] = 'None'\n",
    "    param['train_metric'] = True\n",
    "    param['verbose'] = -1\n",
    "    param['bagging_freq']=5\n",
    "    param['num_class']=3 \n",
    "    param['feature_pre_filter']=False\n",
    "    return param\n",
    "def f1_score_eval(preds, valid_df):\n",
    "    labels = valid_df.get_label()\n",
    "    preds = np.argmax(preds.reshape(3, -1), axis=0)\n",
    "    scores = f1_score(y_true=labels, y_pred=preds, average='macro')\n",
    "    return 'f1_score', scores, True\n",
    "def lossFun(param):\n",
    "    param = getParam(param)\n",
    "    m = lgb.train(params=param,train_set=train_data,num_boost_round=param['num_boost_round'],\n",
    "                          valid_sets=[train_data,valid_data],valid_names=['train','valid'],\n",
    "                          feature_name=features,feval=f1_score_eval,\n",
    "                          early_stopping_rounds=earlyStopping,verbose_eval=False,keep_training_booster=True)\n",
    "    train_f1_score = m.best_score['train']['f1_score']\n",
    "    valid_f1_score = m.best_score['valid']['f1_score']\n",
    "    loss_f1_score = 1 - valid_f1_score\n",
    "    print('训练集f1_score:{},测试集f1_score:{},loss_f1_score:{}'.format(train_f1_score, valid_f1_score, loss_f1_score))\n",
    "    return {'loss': loss_f1_score, 'params': param, 'status': STATUS_OK}\n",
    "\n",
    "features = model_feature\n",
    "train_data = lgb.Dataset(data=X_train[model_feature],label=y_train,feature_name=features)\n",
    "valid_data = lgb.Dataset(data=X_verify[features],label=y_verify,reference=train_data,feature_name=features)\n",
    "\n",
    "best_param = fmin(fn=lossFun, space=spaceParam, algo=tpe.suggest, max_evals=100, trials=Trials())\n",
    "best_param = getParam(best_param)\n",
    "print('Search best param:',best_param)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "经过特征选择和超参数优化后，最终的模型使用为将参数设置为贝叶斯优化之后的超参数，然后进行5折交叉，对测试集进行叠加求平均。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Use 200 features ...\n",
      "the 1 training start ...\n",
      "Training until validation scores don't improve for 100 rounds\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\SOFTWEAR_H\\Anaconda3\\lib\\site-packages\\lightgbm\\engine.py:151: UserWarning: Found `num_boost_round` in params. Will use it instead of argument\n",
      "  warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[100]\tvalid_0's f1_score: 0.894256\n",
      "[200]\tvalid_0's f1_score: 0.909942\n",
      "[300]\tvalid_0's f1_score: 0.913423\n",
      "[400]\tvalid_0's f1_score: 0.917897\n",
      "[500]\tvalid_0's f1_score: 0.920616\n",
      "Early stopping, best iteration is:\n",
      "[456]\tvalid_0's f1_score: 0.920717\n",
      "the 2 training start ...\n",
      "Training until validation scores don't improve for 100 rounds\n",
      "[100]\tvalid_0's f1_score: 0.918357\n",
      "[200]\tvalid_0's f1_score: 0.916436\n",
      "Early stopping, best iteration is:\n",
      "[140]\tvalid_0's f1_score: 0.92449\n",
      "the 3 training start ...\n",
      "Training until validation scores don't improve for 100 rounds\n",
      "[100]\tvalid_0's f1_score: 0.915242\n",
      "[200]\tvalid_0's f1_score: 0.927189\n",
      "[300]\tvalid_0's f1_score: 0.930614\n",
      "Early stopping, best iteration is:\n",
      "[238]\tvalid_0's f1_score: 0.930614\n",
      "the 4 training start ...\n",
      "Training until validation scores don't improve for 100 rounds\n",
      "[100]\tvalid_0's f1_score: 0.901683\n",
      "[200]\tvalid_0's f1_score: 0.912985\n",
      "[300]\tvalid_0's f1_score: 0.916988\n",
      "[400]\tvalid_0's f1_score: 0.92147\n",
      "[500]\tvalid_0's f1_score: 0.921353\n",
      "Early stopping, best iteration is:\n",
      "[411]\tvalid_0's f1_score: 0.922153\n",
      "the 5 training start ...\n",
      "Training until validation scores don't improve for 100 rounds\n",
      "[100]\tvalid_0's f1_score: 0.900975\n",
      "[200]\tvalid_0's f1_score: 0.908373\n",
      "[300]\tvalid_0's f1_score: 0.91384\n",
      "[400]\tvalid_0's f1_score: 0.917567\n",
      "Early stopping, best iteration is:\n",
      "[369]\tvalid_0's f1_score: 0.919843\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0     0.8726    0.9001    0.8861      1621\n",
      "           1     0.9569    0.8949    0.9249      1018\n",
      "           2     0.9586    0.9619    0.9603      4361\n",
      "\n",
      "    accuracy                         0.9379      7000\n",
      "   macro avg     0.9294    0.9190    0.9238      7000\n",
      "weighted avg     0.9385    0.9379    0.9380      7000\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-42-6cbdd079efb6>:88: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  test_['label'] = np.argmax(test_pred, axis=1)\n"
     ]
    }
   ],
   "source": [
    "def f1_score_eval(preds, valid_df):\n",
    "    labels = valid_df.get_label()\n",
    "    preds = np.argmax(preds.reshape(3, -1), axis=0)\n",
    "    scores = f1_score(y_true=labels, y_pred=preds, average='macro')\n",
    "    return 'f1_score', scores, True\n",
    "\n",
    "def sub_on_line_lgb(train_, test_, pred, label, cate_cols, split,\n",
    "                    is_shuffle=True,\n",
    "                    use_cart=False,\n",
    "                    get_prob=False):\n",
    "    n_class = 3\n",
    "    train_pred = np.zeros((train_.shape[0], n_class))\n",
    "    test_pred = np.zeros((test_.shape[0], n_class))\n",
    "    n_splits = 5\n",
    "\n",
    "    assert split in ['kf', 'skf'\n",
    "                    ], '{} Not Support this type of split way'.format(split)\n",
    "\n",
    "    if split == 'kf':\n",
    "        folds = KFold(n_splits=n_splits, shuffle=is_shuffle, random_state=1024)\n",
    "        kf_way = folds.split(train_[pred])\n",
    "    else:\n",
    "        #与KFold最大的差异在于，他是分层采样，确保训练集，测试集中各类别样本的比例与原始数据集中相同。\n",
    "        folds = StratifiedKFold(n_splits=n_splits,\n",
    "                                shuffle=is_shuffle,\n",
    "                                random_state=1024)\n",
    "        kf_way = folds.split(train_[pred], train_[label])\n",
    "\n",
    "    print('Use {} features ...'.format(len(pred)))\n",
    "    #将以下参数改为贝叶斯优化之后的参数\n",
    "    params = {\n",
    "        'learning_rate': 0.05,\n",
    "        'boosting_type': 'gbdt',\n",
    "        'objective': 'multiclass',\n",
    "        'metric': 'None',\n",
    "        'num_leaves': 60,\n",
    "        'feature_fraction':0.86,\n",
    "        'bagging_fraction': 0.73,\n",
    "        'bagging_freq': 5,\n",
    "        'seed': 1,\n",
    "        'bagging_seed': 1,\n",
    "        'feature_fraction_seed': 7,\n",
    "        'min_data_in_leaf': 15,\n",
    "        'num_class': n_class,\n",
    "        'nthread': 8,\n",
    "        'verbose': -1,\n",
    "        'num_boost_round': 1100,\n",
    "        'max_depth': 7,\n",
    "    }\n",
    "    for n_fold, (train_idx, valid_idx) in enumerate(kf_way, start=1):\n",
    "        print('the {} training start ...'.format(n_fold))\n",
    "        train_x, train_y = train_[pred].iloc[train_idx\n",
    "                                            ], train_[label].iloc[train_idx]\n",
    "        valid_x, valid_y = train_[pred].iloc[valid_idx\n",
    "                                            ], train_[label].iloc[valid_idx]\n",
    "\n",
    "        if use_cart:\n",
    "            dtrain = lgb.Dataset(train_x,\n",
    "                                 label=train_y,\n",
    "                                 categorical_feature=cate_cols)\n",
    "            dvalid = lgb.Dataset(valid_x,\n",
    "                                 label=valid_y,\n",
    "                                 categorical_feature=cate_cols)\n",
    "        else:\n",
    "            dtrain = lgb.Dataset(train_x, label=train_y)\n",
    "            dvalid = lgb.Dataset(valid_x, label=valid_y)\n",
    "\n",
    "        clf = lgb.train(params=params,\n",
    "                        train_set=dtrain,\n",
    "#                         num_boost_round=3000,\n",
    "                        valid_sets=[dvalid],\n",
    "                        early_stopping_rounds=100,\n",
    "                        verbose_eval=100,\n",
    "                        feval=f1_score_eval)\n",
    "        train_pred[valid_idx] = clf.predict(valid_x,\n",
    "                                            num_iteration=clf.best_iteration)\n",
    "        test_pred += clf.predict(test_[pred],\n",
    "                                 num_iteration=clf.best_iteration) / folds.n_splits\n",
    "    print(classification_report(train_[label], np.argmax(train_pred,\n",
    "                                                         axis=1),\n",
    "                                digits=4))\n",
    "    if get_prob:\n",
    "        sub_probs = ['qyxs_prob_{}'.format(q) for q in ['围网', '刺网', '拖网']]\n",
    "        prob_df = pd.DataFrame(test_pred, columns=sub_probs)\n",
    "        prob_df['ID'] = test_['ID'].values\n",
    "        return prob_df\n",
    "    else:\n",
    "        test_['label'] = np.argmax(test_pred, axis=1)\n",
    "        return test_[['ID', 'label']]\n",
    "\n",
    "use_train = all_df[all_df['label'] != -1]\n",
    "use_test = all_df[all_df['label'] == -1]\n",
    "# use_feats = [c for c in use_train.columns if c not in ['ID', 'label']]\n",
    "use_feats=model_feature\n",
    "sub = sub_on_line_lgb(use_train, use_test, use_feats, 'label', [], 'kf',is_shuffle=True,use_cart=False,get_prob=False)"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
